Selective Demonstrations for Cross-domain Text-to-SQL
Authors: Shuaichen Chang
Eric Fosler-Lussier
Venue: EMNLP 2023 Findings (2023)
Keywords: text-to-SQL
semantic parsing
in-context learning

Abstract: Large language models (LLMs) with in-context learning have demonstrated impressive generalization capabilities in the cross-domain text-to-SQL task, without the use of in-domain annotations. However, incorporating in-domain demonstration examples has been found to greatly enhance LLMs' performance. In this paper, we delve into the key factors within in-domain examples that contribute to the improvement and explore whether we can harness these benefits without relying on in-domain annotations. Based on our findings, we propose a demonstration selection framework, ODIS, which utilizes both out-of-domain examples and synthetically generated in-domain examples to construct demonstrations. By retrieving demonstrations from hybrid sources, ODIS leverages the advantages of both, showcasing its effectiveness compared to baseline methods that rely on a single data source. Furthermore, ODIS outperforms state-of-the-art approaches on two cross-domain text-to-SQL datasets, with improvements of 1.1 and 11.8 points in execution accuracy, respectively.

Selective Demonstrations for Cross-domain Text-to-SQL.
Shuaichen Chang and Eric Fosler-Lussier
The Ohio State University
[chang.1692, fosler-lussier.1 @ osu.edu
Abstract
information without any demonstrations (Rajku- mar et al., 2022; Liu et al., 2023) or utilize out-of- domain demonstration examples that are annotated NLQ-SQL pairs associated with the databases that are different from the test database (Poesia et al.. 2022; Chen et al., 2023; Pourreza and Rañiii, 2023). However, Rajkumar et al. (2022) and Chang and Fosler-Lussier (2023) have found that the per- formance of LLMs can be significantly improved with annotated in-domain examples serving demonstrations, which are the NLQ-SQL pairs corresponding the test database. Despite the remarkable performance improvements achieved through in-domain annotated demonstrations, ac- quiring such data can be costly, as the annotation process requires SQL professionals. More impor- tantly, annotating examples for each new database diminishes the generalizability of text-to-SQL. ap- plications. These observations naturally raise two questions: (1) Which are the key factors within the in-domain annotated examples that contribute to the performance improvement? and (2) Can we harness these benefits of in-domain demonstrations without relying on in n--ommin annotation?
Large language models (LLMs)) with in-context learning have demonstrated impressive gener- alization capabilities the cross-domain text- to-SQL task, without the use of in-domain notations. However, incorporating in-domain demonstration examples been found to greatly enhance LLMS' performance. In this paper, we delve into the key factors within in-domain examples that contribute to the im- provement and explore whether we can harness these benefits without relying on in-domain an- notations. Based a demonstration selection framework ODIS which utilizes both out-of-domain examples and synthetically generated in in-domain exam- ples to construct demonstrations. By retriev- ing demonstrations from hybrid sources, ODIS leverages the advantages of both, showcasing its effectiveness compared to baseline methods that rely single data source. Furthermore, ODIS outperforms state-of on two cross-domain text-to-SQL. datasets, with improvements of 1.1 and 11.8 points in execu- tion accuracy, respectively.
findings, propose
approaches
1 Introduction
In this paper, we start by investigating the role of the in-domain demonstration examples and then developing new techniques to create demonstra- tions without leveraging in-domain annotations. We assess three aspects within in-domain annota- tions: text-to-SQL. task knowledge and format, in- put NLQ distribution, and output SQL distribution. Our experiments show that SQL distribution plays a pivotal role in in-domain demonstrations. This finding motivates us to synthesize in in-domain data by generating sampled SQL queries. To the best of our knowledge, examples for text-to-SQL. with in-context learning. Furthermore, we introduce a novel demonstration selection framework ODIS which utilizes Out-of- domain Demonstrations and In-domain Synthetic data. By automatically selecting demonstration
Large language models (LLMs), such as GPT-3 (Brown al., 2020), Codex (Chen al., 2021), PaLM (Chowdhery et al., 2022), and LLaMA (Tou- vron et al., 2023), have demonstrated a great ca- pability addressing various language tasks with in-context learning (ICL., which rely on prompts that contain a task instruction and zero or few demonstration examples regarding the task.
Recent studies have evaluated LLMs on the cross-domain text-to-SQL. which translates a nat- ural language question (NLQ) to a SQL query for a new database. Previous work has conditioned LLMs with prompts that solely contain the database
the first to leverage synthetic
'The code for the paper is available at https: //::::///tttt thub. con/shuaichennhaareee Tex Text- SQL.QL.

---

CreatteTablerssleecttooooooo
Using valid SQLite, answer the following questions
Using valid SQLite, answer the following questions for the tables provided Question: Find the number of students in each major. select count(*), najor fron student group by najor; Question: Find the total capacity of all dorns. select sum(student_capacti
tables provided above.
Question: what is the name and nation of the singer who have song having 'Hey' in its name?
select name, country from singer where sang_name like "Hey":
fron dorm;
Question: How many concerts are there in year 2014 or 2815?
TreateTabll+Seleeeeee (concert_singer
select count(*) fron concert year
year = 2015;
Using valid sQLite, answer the following questions far the tables provided above. Question: Which year has nost number of concerts? select
Questian: Which year has most number of concerts? select
Figure 1: An example prompt of 2-shot in-domain text-to-QIL. for database concert_singer Spider dataset (Yu al., 2018). The prompt text of database schema and content, which is constructed using SoottTableeeecctoo. (Chang and Foster-Lussier, 2023), is shown at the beginning of the prompt. Two in-domain demonstrations (highlighted in blue) are pre- sented prior the test question.
Figure An example prompt of 2-shot out-of-domain text-to-SQI. for the test database concert ancert_singer An out-of-domain database dorm_1 with associated demonstrations (highlighted in blue) is presented prior to the test database and question.
examples. Following Chang and Fosler-Lussinn (2023), we utilize leave-one-out split for evalu- ating in-domain demonstrations. Specifically, for given test example, we randomly select demon- strations from the remaining examples associated with the same database, with the condition that lected demonstration examples that do not share the same SQL template as the test example, follow- ing the template-split approach (Finegan- et al., 2018). We also require the selected demon- strations to have different templates following Levy et al. (2022). We repeat all experiments three times to obtain average results. Figure 1 illustrates prompt example that incorporates two in-domain demonstrations for the database concert, senicert_siinnn singer in Spider. Detailed prompt examples can in Ap- pendix A.1. We use Code-davinci-on version of Codex (Chen et al., 2021) as our LLM due to its demonstrated capability on code generation.
examples from both out-of-domain and synthetic in-domain data with SQL-guided retrieval methods, our method consistently outperforms both state- of-the-art models employing fine-tuning (Scholak al., 2021; I; Li et al., 2023a) and in-context learn- ing (Chen al., 2021; Pourreza and Rafiei, 2023) on two cross-domain lext-to-SQL datasets. Our contributions are three folds:
We conduct thorough analysis to examine the impact of different aspects in in-domain demon- strations for text-to-SQL.
We propose demonstration selection framework ODIS that leverages out-of-domain and syntheti- cally generated in-domain demonstrations.
By employing SQL-guided retrieval methods for selecting out-of-domain and synthetic in in-domain demonstrations, ODIS consistently outperforms baselines as well state-of-ther thetth-rrt approaches.
2.2 Effectiveness of In-domain Annotations
2 Analysis of In-domain Demonstrations
Figure illustrates the execution accuracy of Codex Spider and KaggleDBQA. In the zero-shot sce- nario, where no demonstration examples of NLQ- SQL pairs are provided, Codex achieves execution accuracies of 75.7 and 26.8 for Spider and Kag- gleDBQA. However, the utilization of in-domain examples demonstrates a considerable improve- ment in Codex's performance. Notably, as the number of in-domain demonstration examples in- creases, Codex's 's performance continues to enhance. With 30 in-domain annotated examples. Codex's performance is significantly boosted to 84.6 and 71.7 on Spider and KaggleDBQA, respectively.
In this section, we analyze the roles and contri- butions of in-domain annotated demonstrations.
2.1 Experiment Setup
We conduct experiments on two widely-used cross- domain text-to-SQL datasets: Spider (Yu et al., 2018) and KaggleDBQA (Lee et al., 2021). For the Spider dataset, we use its development set, which consists of 20 databases with 1047 exam- ples? 2 As for KaggleDBQA, we employ the com- plete dataset, which contains 8 databases and 272 "The test set of Spider is not publicly available.
While the benefit employing in-domain exam-

---

Zoro-shot
Zero-shot
6q
- 10 Examples
ID Examples
Execution Accuracy
Examples
Accuracy
00D Examples
82
Distribttion
60
Distribution
ID SQL Distribution
ID SQL Distribution
50
80
Execution
80
76
20
0
10
15
20
25
30
0
10
15
20
25
30
Demonstration Examples
Demonstration Examples
(a) Spider
(b) KaggleDBQA
Figure 3: The results of Codex with randomly selected in-domain demonstration examples compared to the zero- shot setting. The x-axis and y-axis represent the number of demonstration examples and the execution accuracy, respectively. represents the experiments utilizing in-domain demonstrations while OOD represents use of out-of-domain demonstrations.
ples in ICL is evident, there still exists limited understanding of the key factors within in-domain demonstrations that contribute to performance en- hancement. To answer this question, we conduct experiments to analyze three aspects of in nn-domain demonstrations: text-to-SQL task knowledge and format, the distribution of in-domain NLQs, and the distribution of in-domain SQL queries. The first aspect pertains to domain- agnostic knowledge, while the latter two specific the domain.
amples. Furthermore, we observe that the perfor- mance gains achieved through in-domain demon- strations tend to converge or even diminish after a certain number of examples have been provided. This finding stands in contrast to the continuous performance improvement observed with the use of in-domain demonstrations. We believe that while Codex is capable of learning the task format and knowledge from in-domain examples, the task for- mat and knowledge alone not sufficient to count for the observed performance gains.
Text-to-SQL task knowledge and format The in-domain examples encompass the task format and knowledge of LLMs, such as the relationship between an NLQ and its corresponding SQL query given database. To investigate the extent to which the performance gains are due to the task format and knowledge, conducted study utilizing an- notated out-of-domain data an alternative source for providing task format and knowledge. In this sctup, the model can learn the correct task format and knowledge from the annotated NLQ and SQL pairs associated with other databases. We randomly select examples from the training set of Spider, which contain 140 databases with 7000 examples. Figure 2 illustrates an example prompt that in- corporates out-of-domain demonstrations. We in- sert databases preceding the test database, each accompanied by 5 NLQ and SQL pairs. We con- ducted experiments varying the value of M, rang- ing from to 6. The results of "OOD Examples" shown in Figure demonstrate that exposure to out- of-domain examples with the same task format and knowledge can indeed enhance the model's perfor- mance. However, the performance gains from these out-of-domain examples are significantly smaller compared to those obtained from in-domain ex-
In addition, we investigate the necessity of task knowledge in the demonstrations by conducting an experiment where we shuffle the NLQs and SQL queries to create mismatched demonstrations. Fig- ure demonstrates that the performance of Codex is significantly reduced when using demonstrations with mismatched NLQs and SQLs compared to the matched NLQs and SQLs. This finding highlights the necessity of including correct task knowledge and format in the demonstrations. Detailed results of this experiment can be found in Appendix A.2.
In-domain input NLQ distribution Besides task knowledge, in-domain examples also present the input NLQ distribution to LLMs. Recent search has indicated that LLMs can benefit from being aware of the input distribution in classifica- tion tasks. Consequently, paring actual inputs with either random output labels (Min et al., 2022) or perturbed outputs (Wang et al., 2022a) has limited impact on the performance of LLMs.
To assess the importance of the input distribu- tion, we substitute the gold SQL queries in the demonstration examples with the predictions gener- ated by Codex the zero-shot setting. This allows the model to be aware of the distribution of in-

---

put NLQs without being exposed the annotated SQLs or any additional knowledge beyond its own predictions. The "ID NLQ Distribution" Figure 3 indicates that, for Spider, providing in-domain NLQ distribution yields some benefits but is not comparable to utilizing complete in in-domain data. Moreover, for the KaggleDBQA dataset, providing in-domain NLQs with seef-perdieted SQL queries does not exhibit any advantage compared to not using any examples. We attribute this observation the lower accuracy of Codex in the zero-shot setting for KaggleDBQA.
to provide accurate task knowledge in addition synthetic data. We propose ODIS: a demonstration selection framework that leverages both Out-of- domain Demonstrations and In-domain Synthetic demonstrations. In Section 3.1 and 3.2, we present our methods for selecting out-of-domain and in- domain data, respectively, within the ODIS frame- work.
3.1 Out-of-domain Demonstration Creation
Recent work has discovered that LLMs sensi- tive the choice of demonstration examples, and LLMs usually benefit from demonstrations that are similar to the test example (Liu et al., 2021a; Wang et al., 2022b; Rubin et al., 2021). Following that, we further hypothesize that the similarity of the output SQL queries between test examples and demonstration examples is more important than input similarity, as found that SQL distribu- tion plays erucial role in the in-domain scenario. Therefore, we propose retrieval strategy SimSQL to retrieve demonstration examples based on the similarity predicted SQL queries.
In-domain output SQL distribution In- domain demonstration examples also serve the pur- pose of revealing the output SQL distribution to LLMs. To assess the significance of this aspect, we employ the same LLM, Codex, to generate synthetic NLQs from oracle SQL queries. In the demonstration examples, we replace the annotated NLQs with these synthetic NLQs while keeping the SQL queries unchanged. This setup allows the LLMs to be exposed to the in-domain SQL distri- bution while remaining unaware of the annotated in-domain NLQQ other inputs beyond its own generated NLQs.
For a test database and a question x, our objective is to retrieve M databases, cach with K NLQ and SQL pairs from a set of annotated out-of-domain examples el,..,,N, where e; (db,, Fi Va. k.))),, representing the database, input NLQ, annotated SQL query, and predicted SQL query by an LLM in the zero-shot scenario, re- spectively. The selected M databases and their associated examples will be presented before the test database and question as shown in Figure 2. Algorithm 1 illustrates our procedure: (1) We first generate an initial SQL prediction ý under the zero-shot scenario (line 1); (2) sort out-of- domain examples based on the similarity between their predicted SQL query d and (line 2); (3) we scan out-of-domain examples from high similarity to low. Once K examples are found a database. the database along with these K examples is se- lected as a demonstration database and examples (line 7-9). The algorithm stops when M databases are selected (line 10-12). We measure the similarity of SQLs with BM25 (Robertson et al., 2009) the SQL keyword and schema tokens in SQL queries.
The results of "ID SQL Distribution" in Fig- ure 3 demonstrate that providing annotated SQL queries with synthetic NLQs can greatly enhance the model's performance on both the Spider and KaggleDBQA datasets. Furthermore, the perfor- mance gains continue to increase with the inclusion of more demonstration examples, aligning with the results obtained using actual in-domain examples.
3 Methods
The experimental results presented in Section 2 demonstrate that the SQL distribution encompassed in in-domain annotated examples plays crucial role in improving performance. However, is im- portant to note that oracle SQL distributions are not available in most cross-domain text-to-SQL ap- plications. To address this limitation, we propose utilizing synthetic SQL queries and NLQs as in- domain demonstrations to capture the in-domain SQL distribution. As the synthetic examples may not always be correct, relying only synthetic ex- amples may potentially contradict the task knowl- edge of text-to-SQL, which could weaken Codex's performance as illustrated in Figure 7. Therefore, to strike a balance between SQL distribution and task knowledge, we leverage out-of-domall data
3.2 In-domain Synthetic Demonstration Creation
In-domain synthetic demonstration selection con- sists of two stages: (1) synthetic data generation, and (2) synthetic data retrieval.

---

Algorithm 1 SimSQL for out-of-domain retrieval Input: zero-shot text-to-SQL model Model, a test database d and question I, and a set of OOD examples OOD = fel, e2, eN} where G = (dz, Ti, Vi. Dz), representing the database, in- put question, gold SQL query, and predicted SQL of Model in the zero-shot setting, respectively. Output: A list of M Demo[d;), where Demo[d] contains K examples associated to d..
ple, questions commonly asked in a flight booking system may not be easily inferred from those in a concert arrangement system. Therefore, instead of expecting to find SQL queries that closely resemble the test question, our focus is on retrieving multiple queries that cover different aspects of the expected SQL query.
To address this, we formulate the problem as maximum coverage problem and adopt a greedy algorithm inspired by the algorithm proposed by Levy et al. (2022). Algorithm 2 outlines the pro- cess of retrieving demonstrations from synthetic in-domain examples: (1) We begin by creating a set of tokens Suncwer that need to be covered, which is initialized with the SQL keyword and schema tokens mentioned in the initial SQL prediction of test question (line 3). (2) We retrieve the synthetic SQLs that have the highest similarity to Suncover. measured with BM25 scores (line 5). The Suncover will be updated by removing the tokens in the re- trieved SQL query and the retrieved example will be added to the demonstration list (line 8 9). We repeat this process until either Suncover becomes empty (line 17) we are unable to retrieve syn- thetic SQL containing any tokens in Suncover (line 14). (3) If the number of selected examples is less than the maximum desired, we iterate through steps (1) and (2) again (line 2).
I: 1 (a)
2: Sort OOD
} in descending order in terms of Simi mirilariie mamilarity(j.ii (.)
3: for e, in OOD do
4: (d,, Vt. Vi. 0s) < Cj
5: if size of Demo[di] < then
6:
Demolds [d:] + Demo[d;] + (d, F. x,,y,) ys) if size of Demold]] = K then
7:
Output + Output + Demold,] end if 10: if size of Output = M then
8:
9:
Break 12: end if
13: end if
14: end for
Synthetic data generation To generate synthetic data, follow previous work to first sample synthetic SQL queries {yi} and then translate SQL queries into natural language questions (*t) (Zhong et al., 2020b; Wang et al., 2021; Zhao et al., 2022). We use SHiP (Zhao et al., 2022) to sam- ple synthetic SQL queries, which extract templates from out-of-domain databases and sample columns and values from the test database to fill the tem- plates. After obtaining synthetic SQL queries, use the Codex to generate corresponding synthetic NLQs, in the same procedure as in our analysis of in-domain SQL distribution.
4 Experiments
4.1 Baseline Methods
Previous Work We compare ODIS with state-of- the-art approaches that either require finetuning on the Spider training set or utilize in-context learn- ing. For the finetuning-based methods, we select SmBoP (Rubin and Berant, 2021) which utilizes RoBERTa-large as the pretrained model, as well as T5+Picard (Scholak et al., 2021), ShiP+Picard (Zhao al., 2022), and RESDSQL (Li al., 2023a), which employ T5-3B (Raffel al., 2020) as the pretrained model.
To improve the quality of synthetic data, we fol- low the previous work on code generation (Zhong al., 2020b; Batra et al., 2021)(zz to add verification process. We use Codex translate the synthetic NLQ z; back SQL iA and filter out the examples that à and y, have different execution results.
For the in-context learning methods, we include the approaches proposed by Rajkumar et al. (2022), Chang and Fosler-Lussier (2023), Lan et al. (2023) which utilize either no demonstrations or randomly selected demonstrations. We also include SYN- CHROMESH (Poesia et al., 2022) and SKILL- KNN (An et al., 2023), which employ similarity- based retrieval methods. Additionally, we se- lect state-of- -fate-of-the-ar the-art approaches LEVER (Ni et al., 2023), Self- tiffDebug (Chen et al., 2023), and DIN-
Synthetic data retrieval While our objective is synthesize SQL queries that align with the or- acle SQL distribution, it is important note that the oracle SQL. distribution often relies on domain- specific prior knowledge that may not be available in the cross-doman text-to-SQP LQL setting. For exam-

---

Algorithm 2 CovSQL for in-domain retrieval Input: zero-shot lext-to-SQL model Model, test database and question æ, and a set of synthetic in-domain examples ID = {e1, e2, .... eN) where Cj = (1i, Vi).
seeds and report the average results. We conducted experiments with both closed-source LLMs, Codex and ChatGPT, in addition to the open-source LLM, CodeLLama, for the final prediction Due to resource constraints, we only employ Codex for retrieving demonstrations.
Output: A list of K examples from DD.
1: 1 - Mo)el(x)
Hyper-parameters For out-of-domain demon- strations, we employ 5 NLQ-SQL pairs for each demonstration database, in line with our experi- ments in Section 2. Determining the number out-of-domain databases and the number of syn- thetic in-domain examples is considered a hyper- parameter selection process. In our final experi- ments for Spider and KaggleDBQA, number of out-of-domain databases to 4 and syn- thetic in-domain examples 5, based the sults obtained from experiments conducted on a randomly-selected subset of 20 databases from the Spider training set.
2: while size Output < do
3: Suncover t tokens in 9
4: while size of Suncover 0 do
5:
ex + argmax Sim( unccover y:) GEID
6:
,²,4²= te
7:
if Sim( Suncuerr, y²) ) > then
8:
u & uncooer tokens in y
set the
9:
Output t Output +
10:
ID + ID - - if' size of Output = K then
11:
12:
Break
end if
else
Break
Results
16:
end if
17:
end while
5.1 Main Results
18:
end while
Table and Table 2 present the execution accuracy of state-of-the-art methods and our proposed ODIS Spider and KaggleDBQA. On Spider, ODIS with Codex achieves execution accuracy of 85.2, surpassing both the state-of-the-at finetuning- based model RESDSQL (Li et al., 2023a) and the in-context learning method Self-Debugging (Chen et al., 2023) by 1.1 points. On KaggleDBQA, ODIS achieves an execution accuracy of 54.8, outper- forming the state-of-the-art model SKILL-KNN (An al., 2023) by 11.8 points. ODIS also demon- strates superior performance compared to baselines that solely rely on out-of-domain or synthetic in- domain demonstrations, regardless of the backbone LLM used. the Spider dataset, ODIS outper- forms these baselines by 3.1 and 3.6 points when using Codex, by 0.8 and 3.0 points with ChatGPT, and by 4.4 and 1.7 points with CodeLlama. Like- wise, on KaggleDBQA, ODIS surpasses the base- lines by 1.5 and 9.6 points when employing Codex. by 7.3 and 19.8 points with ChatGPT, and by 4.4
SQL (Pourreza and Rafici, 2023), which incorpo- rate fixed demonstration examples with intermedi- ate reasoning steps or self-correction procedures.
Ours To demonstrate the effectiveness of ODIS framework, we compare it to baselines that utilize demonstrations solely from out-of-domain data or synthetic in-domain sources. Furthermore, we eval- uate the proposed SQL-guided retrieval strategies SimSQL and CovSQL by comparing them with the Random retrieval strategy, well the SimNLQ approach, which retrieves examples based on the similarity of input NLQs. In line with previous studies (Rubin et al., 2021: Shi et al., 2022), measure similarity using the cosine distance of sentence embeddings encoded through Sentence- BERT (Reimers and Gurevych, 2019).
4.2 Experiments Setup
We utilize the training set of Spider as the pool for selecting out-of-domain demonstrations, following our experiments in Section 2. We generate and filter synthetic in-domain examples, resulting in 1416 ex- amples for Spider and 512 examples for KaggleD- BQA. When the Random retrieval strategy is used, we conduct three repetitions with different random
*e utilize OpenAl APIs code-davinci-e92 and gpt-3.5-tun 5-turbo-16k-8611 for Codex and ChaiGPT. We opt for Codel. -ama-348-inntee
in CodeLlama experiments. "The reported execution accuracy is measured using the Spider official test-suite evaluation (Zhong et al., 2020a), which may yield different values compared to those presented in the original method papers if different evaluation scripts were utilized.

---

Method
LLM
EX
Method
LLM
EX
Previous Work with Finetuning
Previous Work with Finetuning
SmBoP (Rubin and Berant, 2021)
RB
78.0
SmBoP (Rubin and Berant, 2021) T5+Picard (Scholak et al., 2021)
RB
27.2
T54Picard (Scholak et al., 2021)
T3-3B
79.1
T5-3B
29.8
ShiP+Picard (Zhao et al., 2022)
T5-3B
81.4
Previous Work with In-context Learning
RESDSQL+NaIQQL (Li et al., 2023a)
T3-3B
84.1
Zero-shot (Lan et al., 2023)
Codex
23.9
Previous Work with In-context Learning
Few-shot (Lan al., 2023)
Codex
40.4
Rajkumar ct (2022)
Codex
67.0
SKILL- al., 2023)
Codex
43.0
Chang and Fosler-Lussier (2023)
Codex
76.8
This work
LEVER et al., 2023)
Codex
81.9
DIN-SQL (Pourreza and Raflei, 2023)
Codex
75.6
Codex
26.8
DIN-SQL (Pourreza and Rafiei, 2023)
GPT-4
82.8
Zero-shot
ChatGPT
25.7
Self Debuggingg (Chen al., 2023)
Codex
84.1
Codel
18.8
Codex
53.3
This work
Out-of-domain Demo Only
ChatGPT
45.6
Codex
75.7
CodeL
37.9
7ero-shot
ChatGPT
75.7
Codex
45.2
CodeLlama
70.7
Syuthetic In-domain Demo Only
ChatGPT
33.1
Codex
82.1
CodeI.lama
34.9
Out-of-domain Demo Only
ChatGPT 80.7
Codex
$4.8
CodeLlama
75.6
ODIS
ChaiGPT
52.9
Codex
81.5
CodeLlama 42.3
Synthetic In-domain Demo Only
ChatGPT
78.5
Cada
78.3
Table 2: execution accuracy (EX) on the KaggleD-
Codex
85.2
BQA.
ODIS
ChaiGPT
81.5
Code 80.0
observations that the output SQL distribution plays more crucial role than the input NLQ distribution in improving the model's performance.
Table 1: The execution accuracy (EX) on the Spider development The upper section contains models that are finetuned on the Spider training set, while the middle and bottom sections showease previous meth- ods and our proposed methods, which use in-context learning. The column LLM denotes the language model utilized, either the fine-tuning or in-context learning. RB represents RoBERTa-large.
We also explore the use of oracle SQL queries in our experiments. Retrieving examples with the oracle SQL query of the test question results in an additional improvement of 0.8 and 2.9 points in ex- ecution accuracy on the Spider and KaggleDBQA datasets, respectively. We believe that the larger improvement observed attributed to the lower performance of Codex on this dataset in the zero-shot setting. This suggests that employing a better initial model may lead to further enhancements in the performance of the ODIS framework.
KaggleDBQA can be
and 7.4 points with CodeLlama. These results high- light the effectiveness of leveraging both sources of demonstrations within the ODIS framework.
5.2 Analysis
We evaluate our demonstration retrieval methods SimSQL and CovSQL, in comparison to baseline retrieval methods. To specifically analyze and compare the performance of the retrieval meth- ods within each demonstration source, we evaluate the retrieval methods for out-of-domain demonstra- tions without utilizing in-domain synthetic demon- strations, and vice versa.
Method
Spider
Kaggle
Random
78.7
38.0
SimNLQ
81.2
42.6
SimSQL
82.1
53.3
SimSQL (Oracle)
82.9
56.2
Table The execution accuracy of Codex Spider and KaggleDBQA with different out-of- domainn demon- stration retrieval methods.
Out-of-domain retrieval Table 3 presents the comparison among different retrieval methods when retrieving out-of-domain examples. ob- served that retrieving out-of-domain examples with similar predicted queries outperforms random selection and NLQ similarity- selection on both datasets. This finding aligns with our earlier
Synthetic in-domain retrieval Table 4 provides comparison of the in-domain retrieval methods. On the Spider dataset, retrieving synthetic SQL queries that cover different parts of the initial

---

Method
Spider
Kaggle
Method
LLM
Random
77.0
38.2
Previous Work with Finetuning
SimNLQ
79.2
41.5
SmBoP (Rubin and Berant, 2021) T5+Picard (Scholak et al., 2021)
RB T5-3B
58.1
65.0
SimSQL
79.7
45.2
This work
CovSQL
45.2
Zero-shot
Codex
65.3
Out-of-domain Demo Only
Codex
70.9
CovSQL (Oracle)
81.8
48.9
Synthetic In-domain Demo Only
Codex
70.9
CovSQL (In-domain Annotation)
82.4
52.9
ODIS
Codex
73.4
Table 4: The execution accuracy of Codex on Spider and KaggleDBQA with different synthetic in-domain demonstration retrieval methods.
Table 5: The execution accuracy (EX) of Codex on the Dr. Spider NIQQ post-perturbation sets.
trieving annotated in-domain examples typically necessitates large amount of in-domain annota- tions, which are usually not available.
SQL predictions outperforms the retrieval methods based on the similarity of either NLQs or initial SQL predictions. On the KaggleDBQA dataset, both the CoverSQL and SimSQL methods achieve the same results, outperforming the random selec- tion and NL---smmilarity-bae based methods. Regard- ing the experiments with oracle SQL queries, again, we observe larger improvement on the KaggleD- BQA dataset compared to Spider which may be attributed to the lower performance of Codex on KaggleDBQA.
order to assess the robustness of our proposed ODIS framework, we conduct a study to compare ODIS with the baseline supervised-leaming and in-context-learning meth- ods Dr. Spider (Chang et al., 2023), an evalua- tion benchmark for text-to-SQUL robustness. We opt for the NLQ post-perturbation sets in Dr. Spider, these have been identified as the most challenging for LLMs with in-context learning (Chang et al.. 2023). The results, presented in Table 5, reveal that ODIS with Codex achieves an execution ac- curacy of 73.4 4 on NLQ perturbations, surpasses the supervised learning methods and the Codex with either demonstrations.
Robustness Evaluation
Impact of synthetic data To assess the impact of synthetic data quality in ODIS framework, we com- pare two SQL synthesizing methods SHiP (Zhao et al., 2022) and GAZP (Zhong et al., 2020b). Both methods extract the templates of SQL queries from the Spider training set and fill the template with the database schema and content with new database, however, SHiP leverages a schema-wett weiihted sam- pling to make the synthetic SQL queries more re- alistic by controlling the relationship of sampled columns. When replacing our synthetic data gen- erated through SHiP to GAZP, we find the perfor- mance with Codex has dropped from 81.5 to 79.5 on Spider and from 45.2 to 33.8 on KaggleDBQA. This indicates that the quality of synthetic data is crucial in the ODIZ framework.
synthetic in-domain
Related Work
Text-to-SQL with in-context learning Previous studies have explored text-to-SQL. with in-context learning from various perspectives. Rajkumar et al. (2022) and Chang and Fosler-Lussier (2023) inves- tigated effective approaches to represent databases in prompts. Other studies have explored the incor- poration of intermediate reasoning steps enhance the performance of LLMs (Chen et al., 2023; Pour- reza and Rafiei, 2023; Tai et al., 2023).
Additionally, we evaluate the performance of retrieving from synthetic in-domain examples com- pared to retrieving from actual annotated in-domain examples in Spider and KaggleDBQA with the CovSQL retrieval method. The performance gap in Table 4 between them indicates that refining the data synthesizing method could potentially yield even greater enhancements within the ODIS frame- work. It is worth noting that this experiment is designed to illustrate the limitations of synthetic examples compared to annotated examples. Re-
For demonstration retrieval, prior work has ex- plored different strategies for retrieving out-of- domain examples based on similarity (Poesia et al., 2022; Rubin et al., 2021) and diversity (Levy et al., 2022; Su et al., 2022). Concurrent with work, Nan et al. (2023) suggested retrieving out- of-domain demonstrations based on the initial SQL predictions, which aligns with our retrieval method. Additionally, execution results have been used in the SQL generation process to verify the gen-

---

erated SQL queries (Chen et al., 2023; Ni al.. 2023; Nan al., 2023; Guo et al., 2023). These approaches often require multiple LLM calls and SQL executions to accomplish majority voting or self-correction. It is worth noting that the database execution time is crucial consideration in realistic applications, as highlighted by Li et al. (2023b). In contrast, ODIS with our proposed retrieval strategy only requires two LLM calls (one for initial SQL. prediction and one for final generation) and elimi- nates the need for any database execution, making it a more efficient solution for text-to-SQL tasks.
selection among them. Additionally, our retrieval approach relies on the predictions of Codex in the zero-shot scenario. It worth exploring utiliz- ing eegher-performance initial text-to-SQL SQL mod- els to further enhance the performance, as demon- strated in Section 5.2 through the use of oracle SQL queries. We believe the effectiveness of the ODIS framework will encourage further advancements in data synthesis methods and demonstration retrieval techniques.
Considering the proven effectiveness of utiliz- ing hybrid data sources, comprising out-of-domain examples and synthetic in-domain examples as in- context demonstrations, we believe that ODIS can be extended to few-shot, parameter-efficient fine- tuning by leveraging these hybrid data sources (Hu et al., 2021; :; Dettmers et al., 2023). We leave the exploration of parameter-cffiiient fine-tuning with synthetic in-domain examples for future rescarch.
Data synthesis for Text-to-SQL Due to the high cost of acquiring annotated text-to-SQL data, pre- vious research has explored the use of synthetic data as an alternative approach to enhance the performance of cross-domain text-to-SQL models. These works primarily focused on data augmen- tation, such as generating additional examples for the training databases (Guo et al., 2018; Yu et al., 2021; Wang et al., 2021; Wu et al., 2021; Yang et al., 2021; Liu et al., 2021b; Zhao et al., 2022). Zhong et al. (2020b) first synthesized data for test databases and adapted pretrained text-to-SQL. mod- els on the synthetic data associated with the test database before inference. In a similar vein, ODIS leverages synthetic data for the test database but uses with in-context learning rather than directly fine-tuning models on it.
Limitations
We hope the effectiveness of ODIS will draw at- tention to the potential benefits of incorporating hybrid demonstrations synthetic and annotated data in the text-to-SQL task in in-context learning. As the first study exploring this approach, we exam- ined the retrieval methods separately for these two types of demonstrations. However, as mentioned in Section 7, exploring more advanced retricval methods to blur the boundary of out-of-domin and synthetic in-domain examples is a direction for future work.
7 Conclusion and Future Work
In this work, we delve into the analysis of crucial aspects of in-domain demonstrations and identify the SQL distribution as the key factor. Inspired by our findings, we propose a novel demonstration selection framework, ODIS, which harnesses the power of both out-of-domain demonstrations and synthetic in-domain examples using SQL-guided retrieval. The remarkable performance across dif- ferent backbone large language models demon- strates the effectiveness of our proposed frame- work, compared to both bascline and state-of-the- art methods.
We conducted our experiments with both closed- source Codex 5 and ChatGPT and open-source CodeLlama the large language models. Recent research has shown that GPT-4 consistently outper- forms Codex in the text-to-SQL. generation when employing various prompt strategies (Pourreza and Rafiei, 2023). Considering our budget constraints, we leave the ODIS framework with other large lan- guage models as a topic for future investigation.
While our proposed method involves two calls to L.MSs and no SQL execution during the generation process, it does require an offline process for syn- thetic example generation. We acknowledge that this process adds an additional step to the pipeline. However, we believe that can be seamlessly inte- grated with the database deployment process. By
While ODIS serves as a general framework that can be employed with various demonstration re- trieval methods, our experiments in this study uti- lize separate retrieval methods for out-of-domain and in-domain demonstrations. We regard it as future work to explore a unified retrieval strategy that breaks the boundary between and synthetic in-domain data, enabling automatic
Coodex currently free for research, with API at https://peemi..comaag For commercial use, Codex avail- able at https://azure. http://auurrmmmrooott.o.moo ciacrrooot.com. 'ChatGPT API is available at https//lpeeai..om/app..

---

incorporating the synthetic example generation step during the database deployment phase, users query- ing the database will not experience any noticeable delays or disruptions.
Ethics Statement
We acknowledge the importance of the ACL Ethics Policy and agree with it. The objective of text- to-SQL systems
provide non-expert database users with the ability to query databases using natu- ral language. Our research in this paper focuses on improving the generalization capabilities of text- to-SQL models. While the benefit of in-domain annotated demonstrations is evident, the significant cost associated with such annotation should not be ignored in real-world applications. Therefore, we propose method that combines both out-of- domain examples and synthetic in-domain data to enhance models' generalization and reduce reliance on costly in-domain annotations. We believe that this approach can improve the performance and us- ability of text-to-SQL. systems, making them more accessible and practical for wider range of appli- cations.
Moreover, is important consider the cost LLMs and SQL execution in realistic scenarios. Our proposed method employs two calls to LLMs without requiring any SQL execution during the SQL decoding process. This design choice aims to optimize the cost and user experience compared to other approaches that involve multiple LLM calls and SQL executions.
A Akknoweedmeents
We would like to thank Michael White, Micha El- sner, the OSU SLaTe lab, as well as the anonymous reviewers for their valuable feedback on this work.
Shuaichen Chang is supported in part by the Ama- zon Post Internship Fellowship.

---

References
Daya Guo, Yibo Sun, Duyu Tang, Nan Duan, Jian Yin. Hong Chi, James Cao, Peng Chen, and Ming Zhou. 2018. Question generation from sql queries improves neural semantic parsing, arXiv preprint arXiv:1888.06304
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian Guang Lou. 2023. Skill-based few-shot selection for in-context learning. arXiv preprint arXiv:2305. 14200
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang. and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.0660
Soumya Batra, Shashank Jain, Peyman Heidari, Ankit Arun, Catharine Youngs, Xintong Li. Pinar Donmez, Shawn Mei, Shiunzu Kuo, Vikas Bhardwaj, et al. 2021. building adaptive acceptability classifiers for neural nlg. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 682-697.
Wuwei Lan, Zhiguo Wang, Anuj Chauhan, Henghui Zhu, Alexander Li, Jiang Guo, Sheng Zhang. Chung- Wei Hang. Joseph Lilien, Yiqun Hu, et al. 2023. Unite: A unified benchmark for text-to-sq evaluation. arXiv preprint 2rX.v:2305.162:
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggledIqa: Realistic eval- uation of text-to-sql parsers. arXiv preprint arXiv:2106.11555.
Shuaichen Chang and Eric Fosler-Lussier. 2023. How to prompt Ilss for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings. arXiv prepriat arXiv:2305.1 188533
Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800.
Shuaichen Chang. Jun Wang. Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang. Jiarong Jiang. Joseph Lilien, et al. 2023. Dr. spider: A diagnostic evaluation bench- mark towards text-to-sql robustness. arXiv preprint arXiv:2301.08883
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen, 2023a. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. arXiv preprint arXiv:2302.05965.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang. Bowen Li. Bailin Wang. Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023b. Can 1lm already serve as a database interface? a big bench for large- scale database grounded text-to-sqls, arXiv preprint arXiv:2305.037/
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. 2023. A comprehensive evaluation of chat- gpt's zero-shot text-to-sql capability. arXiv preprint arXiv:2303. 13547.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05//
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a. What makes good in-context examples for gpt-3? arXiv preprint 101.022001.06804.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi. Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2021b. Tapex: Table pre-training via learning a neural sql executor. arXiv preprint 107.v2210..07653.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zeulemoyer. 2023. Qlora: Efficient finetuning of quantized llss. arXiv preprint 2iXiv:2305.1 14314.
Sewon Min, Xinxi Lyu, Ari Holtzman. Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? arXiv preprint arXiv:2202.1282 12837.
Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-sql evaluation methodology. arXiv preprint 1oXiv:1806.0902)
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Ennhancing few-shot text- to-sql capabilities of large language models: A study on prompt design strategies. arXiv preprint 2rXiv:2305.12586
Chunxi Guo. Zhiliang Tian, Jintao Tang, Pancheng Wang, Zhihua Wen, Kang Yang, and Ting Wang. 2023. A case-based reasoning framework for adap- tive prompting in cross-domain text-to-sql. arXiv preprint 23xh:2204413300)

---

Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoy- anov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. 2023. Lever: Learning verify language- to-code generation with execution. arXiv preprint arXiv:2302008668
Chang-You Tai, Ziru Chen, Tianshu Zhang, Xiang Deng, and Huan Sun. 2023. Exploring chain-of of-thu style prompting for text-to-sql. arXiv preprint arXiv:2305. 14215.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fiic Hambro, Faisal Azhar, et 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397/
Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Ti- wari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code gen- eration from pre-trained language models. arXiv preprint arXiv:2201.1 1122.
Mohammadreza Pourreaa and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint arXiv:2304.11015.
Bailin Wang. Wenpeng Yin, Xi Victoria Lin, and Caim- ing Xiong. 2021. Learning to synthesize data for semantic parsing. arXiv preprint arXiv:2104.058:
Boshi Wang. Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a. Towards understanding chain-of- of-thoughh prompting: An empirical study of what matters. arXiv preprint arXiv:2212.1000:
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li. Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified textt trans- former. J. Mach. Learn. Res., 21(140):1-67.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022b. Training data is more valuable than you think: A simple and effective method by retrieving from training data. arXiv preprint orXiv:2203.087
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabil- ities of large language models. arXiv preprint arXiv:2204.0044
Kun Wu, Lijie Wang. Zhenghua Li, Ao Zhang. Xinyan Xiao, Hua Wu, Min Zhang, and Haifeng Wang. 2021. Data augmentation with hierarchical sql-to- question generation for cross-domain text-to-sql pars- ing, arXiv preprint 10Xiv:2103.00 02227
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert. Sentence embeddings using siamese netw-ke.works. arXiv preprint arXiv: 10084.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends in Information Re- trieval, 3(4):333-389
Wei Yang, Peng Xu, and Yanshuai Cao. 2021. Hier- archical neural data synthesis for semantic parsing. arXiv preprint
121.02212..
Ohad Rubin and Jonathan Berant. 2021. Smbop: Semi- autoregressive bottom-up semantic parsing. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311-324.
Tao Yu, Chien- ShennS Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir R Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar- auammar-augmented pre-training for table semantic parsing. In ICLR.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint 10x9iv1809.0000 08887.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.00
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-99919 99019
Yiyun Zhao, Jiarong Jiang, Yiqun Hu, Wuwei Lan, Henry Zhu, Anuj Chauhan, Alexander Li, Lin Pan, Jun Wang. Chung-Wo Wei Hang. al. 2022. Impor- tance of synthesizing high-quality data for text-to-sql parsing, arXiv preprint arXiv:2212.0878.
Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2022. XricI: Cross-lings Eross-lingual strrieval-augmented in-context learning for cross-lingual text-to-sql semantic pars- ing. arXiv preprint arXiv:2210. 13693.
Ruiqi Zhong, Tao Yu, and Dan Klein. 2020a. Semantic evaluation for text-to-sql with distilled test suites. arXiv preprint arXiv:2010.02840.
Hongjin Su. Jungo Kasai. Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selec- tive annotation makes language models better few- shot learners. arXiv preprint arXiv:2209.0197
Victor Zhong, Mike Lewis, Sida I Wang, and Luke Zettlemoyer. 2020b. Grounded adaptation for zero- shot executable semantic parsing. arXiv preprint arXiv:209..0399

---

A Appendix
A.1 Prompt Construction
For the Spider dataset (Yu et al., 2018), we utilize the CreattelbbleSelectooll (Chang and Fosler- Lussier, 2023) prompt construction to create the prompt text for databases. In the KaggleDBQA dataset (Lee al., 2021), each database is accom- panied by a database document that provides de- scriptions of the columns the database. We incor- porate the database document as comments within the CreateTableSelecttoe prompt. Figure 4 and 5 provide examples of the database prompt text for Spider and KaggleDBQA, respectively.
A.2 Necessity of Text-to-SQL Task Knowledge
Figure shows prompt example with mis- matched NI.Qs and SQL. queries as demonstra- tions. this example, the first NLQ corresponds to the second SQL and the second NLQ corresponds to the first NLQ. Figure 7 shows that the perfor- mance of Codex is significantly reduced when us- ing demonstrations with mismatched NLQs and SQLs compared to the matched NI.Qs and SQLs. This finding highlights that it is important to main- tain the correct task knowledge the demonstra- tions, which inspires us to leverage out-of-domain demonstrations along with synthetic in-domain ex- amples.

---

int, create table stadium
stadiun_id int,
location text.
nane text, capacity int, int,
lowest int,
prinary key (stadium_id) ):
Columns in stadiun and distinct examples in each column:
stadium_id id: 1,
location: "Raith Ravers". "Ayr United", "East Fife";
nane: "Stark's Park* "Sonerset Park", "Bayview Stadiun"; capacity: 10104, 11998, 2000;
highest: 4812, 2363. 1988;
lowest: 1294. 1057. 533:
average: 2186, 1477. 864; */
create table singer singer_is int,
nane text
country song_nane text. song-release_yenr text, age int, is_nale bool,
prinary key (singer_id)
):
Columns in singer and 3 distinct examples in each column: singer_id: 1, 2, 3;
name: "Joe Sharp", "Timbaland', "Justin Brown"; country: "Netherlands" "United States", "France"; song_name: "You", "Dangerous", "Hey Oh"; 1992, 2008, 2013;
age: 52, 32, 29; is_male: "F", "T"; +/
create table concert concert_id int.
concert_name text,
thene text, stadium_id text, year text, prinary key (concert_id), foreign key (stadium_id) references (stadiunn)))
/=
Columns in concert and distinct examples in each column: 2, 3;
concert_id: 1,
"Auditions", "Super bootcamp", "Home Visits"; theme: "Free choice", "Free choice "Bleeding Love"; stadium id: 1, 2, 10;
year: 2014, 2015; */
create concert_id id int, singer_id text,
singer_in_concen
prinary key (concert_id, singer_id).
foreign key (cancert_id) references concert (cancert_ foreign key (singer_id) references singer (singer_id) /
Columns in inngerrin_conertt and 3 distinct examples in each column: concert_id: 1, singer_id: 2, 3, 5; */
Figure 4: An database prompt example of for the concert_singer database in Spider.

---

CreateTable+SelectCoo (Studen (aascoor)
table finrev
fed_17 lan_ev_fed_1i_
state_code integer,
the state cade of the finrev_fed_17
idcensus integer, -- the idcensus of the finrey_fed_17 district text,
the school district nces_id text, -- the nces id of the fidrey_fed_17 yr_data integer.
finrev_fed_17
year t_fed_rev integer, total federal revenue through the state to each school district. c14 integer, federal c25 integer federal revenue through the state child nutrition a ):
of fidrev_fed_17
through the state- title 1 child left behind act).
Columns in finrev_fed_1 and 3 distinct examples in each column:
state_code: 33, 5, 14;
idcensus: 3320310013011 301000000000 5501905909000 1450161 5800000; school school_district district: "NEW YORK CITY SCHOOL DISTRICT", *LOS ANGELES UNIF SCH DIST", "CITY OF CHICAGO SCHOOL DISTRICT 299";
nces_id: 3620580, 622710, 1709930;
yr_data: 17; r_ved_rev: 2061297, 1146298. 783943; c14: 956851, ,
376182, 290912;
c25: 439209, 390711, 280517; */
create table ndecoreexcel .math_ggadeeee year integer,
year the state text, the state of the matoreeecel_aath.eeeee all_students text, the all students af the ndecoreexcel_math_.e average_scale_ integer the average scale score of the mdecoreexcel_math.ggne
and 3 distinct examples in each column: state: "National", "Alabama", "Alaska"; all_students: "Ali students": average_scale 283, 268, 277; */
Columns in nattoreeecee__athhggrre year:
create table finrev_fed_key.. state_code code
the state code of the finrev finrev_fed_ken key_k state text, the state of the finrev_fe__key #_records text the nunber of records of the finrev_fed_key_11 );
/
Columns finrev fer_ke__ state_code 1, 2. 3; "Alabama" " "Alaska", "Arizona";
distinct examples column:
#_records: 137, 54, 235; */
Figure 5: An database prompt example of for the StudentMathScore database in KaggleDBQA.

---

Using valid sQLite, answer the following questions for the tables provided above
Question: what is the and nation the singer who have a song having 'Hey' in its name?
select count(*) from concert where year 2014 year 2015;
Question: How many concerts are there in year 2014 or
select name, country from singer where song_name like 'Hey';
Question: Which year has most number concerts?
select
An example prompt of 2-shot in-domain text- to-SQL using mismatched demonstration examples.
Figure 6:
85
Zero-shol Examples
Exccution Accuracy
80
Examples
75
70
60
85
45
Demonstration Examples
(a) Spider
Zara shot
70
1D Examples
Execution Accuracy
Mismatch
80
50
40
30
Demonstration Examples
(b) KaggleDBQA
Figure The results of Codex with real in-domain demonstration examples compared to mismatched ex- amples. ID Examples represents annotated in-domain demonstrations while Examples Misttach repre- sents the NLQ and SQL pairs are mismatched. The x-axis and y-axis represent the number of demonstra- tion examples and the execution accuracy, respectively.


Reviews:
[{"review_id": "UM5kNaQxAa", "reviewer": "Reviewer_cHQZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our approach novel and effective.  We also thank the reviewer for their helpful comments and we have made a response to each of their comments.\n\n\n> I suggest that you should include experiments with GPT-3.5, which is a more affordable option. This would provide a more comprehensive evaluation of your proposed approach.\n\nWe highly value the feedback and agree that experiments on other LLMs would indeed underscore the generalizability of our approach. The table below contains the experimental results with two strong LLMs: OpenAI GPT-3.5 (gpt-3.5-turbo-16k) and an open-sourced model Code Llama [1] (CodeLlama-34b-Instruct). Both the out-of-domain demonstration only method and synthetic in-domain demonstration only method outperform the zero-shot setting on two datasets, with GPT-3.5 and Code Llama. Notably, the ODIS approach shows even more improvements in comparison to both the in-domain only and out-of-domain only methods. Due to our limited budget, we did not undertake a hyperparameter search for the experiments with GPT3.5 and Code Llama. Nevertheless, the current results already illustrate the generalizability of our approach.\n\nAs a result, we believe the remarkable performance of ODIS with both closed-source and open-sourced LLMs on two cross-domain text-to-SQL datasets is able to demonstrate the generalizability of our proposed approach.\n\n\n|Method|LLM|Spider|Kaggle|\n|---|---|---|---|\n|Zero-shot|GPT-3.5|75.7 | 25.7|\n|Out-of-domain Only|GPT-3.5| 80.7| 45.6|\n|Synthetic In-domain Only|GPT-3.5|78.5 | 33.1|\n|ODIS|GPT-3.5|**81.5** |**52.9**|\n||||\n|Zero-shot|Code Llama|70.8 | 18.8|\n|Out-of-domain Only|Code Llama| 76.0| 35.7|\n|Synthetic In-domain Only|Code Llama|78.0 |34.9|\n|ODIS|Code Llama|**78.7** |**40.4**|\n\n\n\n> The paper lacks experiments on the Spider test set. Without experiments on this dataset, it is difficult to evaluate the generalizability of your proposed approach\n\nOur proposed approach involves a preprocessing stage to generate synthetic examples with LLMs which is currently not supported by the Spider evaluation. We are contacting the Spider authors to provide an evaluation on the hidden test set. We would kindly point out that many other studies have also conducted experiments using the Spider development set [2-7], as opposed to the hidden test set. Moreover, it is important to note that the hyperparameters are selected based on a subset of the Spider training set for all experiments on the Spider development set and KaggleDBQA.\nAs a result, we believe the remarkable performance of ODIS with three LLMs (Codex, GPT-3.5, and Code Llama) on the Spider development set and the KaggleDBQA dataset is able to demonstrate the generalizability of our proposed approach.\n\n\n> The examples provided in the appendix are not detailed enough, and it is difficult to find the corresponding prompts for the different methods in Figure 3.\n\nWe appreciate the feedback and we will add a full prompt example for each experiment in the appendix in the revised version. The code will be released as well.\n\n>  I suggest that you further develop your approach to provide more significant contributions to the field.\n\nWe appreciate the feedback. We would like to highlight the novelty and contribution of our work here.\n\nOur work begins with a deep analysis of the effectiveness of gold in-domain demonstrations, where we discovered the SQL distribution within the in-domain data plays a pivotal role. While the insight may not be surprising to the reviewer, such analysis was missed in the field of text-to-SQL before our work and we believe our findings would impart valuable insights to future research in the field of text-to-SQL with in-context learning.\n\nFurthermore, we were also the first to address the cross-domain text-to-SQL challenge with hybrid sources of demonstrations (out-of-domain examples and synthetic in-domain examples) for in-context learning, in contrast to previous efforts that solely focused on retrieving out-of-domain examples. Our empirical findings, substantiated by comprehensive experiments involving both open-sourced and closed-sourced LLMs on the Spider and KaggleDBQA datasets, effectively show the effectiveness of our approach.\n\nGiven the remarkable results of our work, we believe it is a good direction to enhance LLMs’ capability on text-to-SQL with the hybrid source of demonstrations, which would stimulate future works on refining data synthesizing methods and retrieval methods.\n\n[1] Rozière, Baptiste, et al. \"Code Llama: Open Foundation Models for Code.\" arXiv preprint arXiv:2308.12950 (2023).\n\n[2] Wang, Lihan, et al. \"Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing.\" Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.\n\n[3] Ni, Ansong, et al. \"Lever: Learning to verify language-to-code generation with execution.\" International Conference on Machine Learning. PMLR, 2023.\n\n[4] Shaw, Peter, et al. \"Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n\n[5] Qin, Bowen, et al. \"SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.\n\n[6] Poesia, Gabriel, et al. \"Synchromesh: Reliable Code Generation from Pre-trained Language Models.\" International Conference on Learning Representations. 2021.\n\n[7] Pourreza, Mohammadreza, and Davood Rafiei. \"Din-sql: Decomposed in-context learning of text-to-sql with self-correction.\" arXiv preprint arXiv:2304.11015 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a novel prompt approach that leverages in-domain NLQs and SQLs to improve the performance of LLMs on the cross-domain text-to-SQL task.\n2. The authors conduct extensive experiments on two benchmark datasets, Spider and KaggleDBQA, to evaluate the effectiveness of prompts. The results show that new prompts significantly outperforms existing methods and achieves state-of-the-art performance on both datasets.", "reasons_to_reject": "1. The paper lacks experiments on the Spider test set. Without experiments on this dataset, it is difficult to evaluate the generalizability of your proposed approach.\n2. While I understand that GPT-4 is expensive, I suggest that you should include experiments with GPT-3.5, which is a more affordable option. This would provide a more comprehensive evaluation of your proposed approach.\n3. The examples provided in the appendix are not detailed enough, and it is difficult to find the corresponding prompts for the different methods in Figure 3. I suggest that you provide more detailed examples to help readers better understand your approach.\n4. While your proposed approach provides some evidence for in-context learning, I did not find any surprising or novel conclusions in your paper. I suggest that you further develop your approach to provide more significant contributions to the field.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on the problem of cross-domain text-to-SQL, which translates a natural language question to a SQL query. The authors propose a novel prompt approach that leverages in-domain NLQs and SQLs to improve the performance of codex on this task. The main contributions of the paper are as follows:\n1. The authors propose a novel prompt approach that leverages in-domain NLQs and SQLs to improve the performance of LLMs on the cross-domain text-to-SQL task.\n2. The authors conduct extensive experiments on two benchmark datasets, Spider and KaggleDBQA, to evaluate the effectiveness of prompts. The results show that new prompts significantly outperforms existing methods and achieves state-of-the-art performance on both datasets.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xultd3TrAI", "reviewer": "Reviewer_TqYu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our work easy to understand and providing considerable improvement over baselines. We also thank the reviewer for their helpful comments and we have made a response to each of their comments.\n\n\n> The novelty seems limited. Using similarity to retrieve demonstrations is not a new thing.\n\nWe would like to highlight the novelty and contribution of our work here.\n\nOur work begins with a deep analysis of the effectiveness of gold in-domain demonstrations, where we discovered the SQL distribution within the in-domain data plays a pivotal role. Such analysis was missed in the field of text-to-SQL before our work and we believe our findings would impart valuable insights to future research in the field of text-to-SQL with in-context learning. Furthermore, we were also the first to address the cross-domain text-to-SQL challenge with hybrid sources of demonstrations (out-of-domain examples and synthetic in-domain examples) for in-context learning, in contrast to previous efforts that solely focused on retrieving out-of-domain examples. \n\nMoreover, even retrieving similar examples as demonstration is not a new thing, the specific approach would be largely different in various tasks/settings. For example, synthetically generated in-domain data may not contain similar examples to each specific test example. Therefore, our proposed coverage-based retrieval outperforms traditional similarity-based retrieval by retrieving partially similar examples (shown in Table 4).\n\n\n> The setting is not justified. Manually annotating 10 ~ 30 in-domain examples for in-context learning seems not a big deal compared to collecting tens of thousands of examples.\n\n\nThe cross-domain text-to-SQL setting has been a prevalent topic for a long time, encompassing both supervised-learning methods [1-3], and in-context-learning methods [4-6]. Methods in this setting focuses on generalizing to a new database without requiring any data associated with the new database. This setting allows a non-SQL-expert user to create a natural language interface to database (NLIDB) system for any new databases. While annotating 10-30 examples for an SQL expert is not hard, we want to note that annotating 10-30 examples for each new database is extremely costly. For example, the Spider development set has 20 databases. Annotating 10-30 examples for each would require 200-600 SQL annotations. This is the reason that we propose to use pre-existing databases and examples (Spider training set) and synthetically generated examples.\n\n> Codex is a close-sourced deprecated model. I highly encourage the authors to justify the conclusion on open-sourced models like T5 and LLaMA.\n\n\nTo demonstrate the generalization of our proposed framework, we also present a study with two strong LLMs: OpenAI GPT3.5 (GPT-3.5-turbo-16k, a.k.a ChatGPT), and Code Llama (CodeLlama-34b-Instruct, the best open-sourced LLMs for code tasks). The results are presented below. Both the out-of-domain demonstration only method and synthetic in-domain demonstration only method outperform the zero-shot setting on two datasets, with GPT-3.5 and Code Llama. Notably, the ODIS approach shows even more improvements in comparison to both the in-domain only and out-of-domain only methods. Due to our limited budget, we did not undertake a hyperparameter search for the experiments with GPT3.5 and Code Llama. Nevertheless, the current results already illustrate the generalizability of our approach. \n\n\nAs a result, we believe the remarkable performance of ODIS with both closed-source and open-sourced LLMs on two cross-domain text-to-SQL datasets is able to demonstrate the generalizability of our proposed approach.\n\n\n|Method|LLM|Spider|Kaggle|\n|---|---|---|---|\n|Zero-shot|GPT-3.5|75.7 | 25.7|\n|Out-of-domain Only|GPT-3.5| 80.7| 45.6|\n|Synthetic In-domain Only|GPT-3.5|78.5 | 33.1|\n|ODIS|GPT-3.5|**81.5** |**52.9**|\n||||\n|Zero-shot|Code Llama|70.8 | 18.8|\n|Out-of-domain Only|Code Llama| 76.0| 35.7|\n|Synthetic In-domain Only|Code Llama|78.0 |34.9|\n|ODIS|Code Llama|**78.7** |**40.4**|\n\n\n\n\n\n\n\n[1] Rubin, Ohad, and Jonathan Berant. \"SmBoP: Semi-autoregressive Bottom-up Semantic Parsing.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n[2] Scholak, Torsten, Nathan Schucher, and Dzmitry Bahdanau. \"PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[3] Li, Haoyang, et al. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n\n[4] Poesia, Gabriel, et al. \"Synchromesh: Reliable Code Generation from Pre-trained Language Models.\" International Conference on Learning Representations. 2021.\n\n[5] Li, Jinyang, et al. \"Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.\" arXiv preprint arXiv:2305.03111 (2023).\n\n[6] Liu, Aiwei, et al. \"A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability.\" arXiv preprint arXiv:2303.13547 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach demonstrates considerable improvement over baselines.\nThe paper is easy to understand.", "reasons_to_reject": "1. The novelty seems limited. Using similarity to retrieve demonstrations is not a new thing.\n2. The setting is not justified. Manually annotating 10 ~ 30 in-domain examples for in-context learning seems not a big deal compared to collecting tens of thousands of examples.\n3. Codex is a close-sourced deprecated model. I highly encourage the authors to justify the conclusion on open-sourced models like T5 and LLaMA.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work studies how to select demonstrations for text-to-SQL generation using in-context learning. The authors assume that in-domain examples to hard to obtain and mimic in-domain examples by utilizing retrieved out-domain examples and synthesized in-domain examples. Codex is used for all experiments and improvement is demonstrated on Spider and KaggleDBQA.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OfizDftlsa", "reviewer": "Reviewer_E6z7", "summary": "", "questions": "1.\tFor Weakness 1, is the difference due to different demonstration selection methods? Will the finding hold using the same selection method.\n2.\tWhat’s the number of demonstrations used in each method in Table 1 and does the number of demos affect the results? If so, an analysis on this would be valuable.\n\nMinor question: DIN-SQL method shows using GPT-4 outperforms Codex by 7% accuracy. Will applying ODIS to GPT-4 achieve even higher accuracy? Providing a discussion based on a subset is ok if the budget is limited.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer provided a nice summary of our work and found both of our analysis and proposed method valuable and novel. We also thank the reviewer for their helpful comments and we have made a response to each of their comments.\n\n\n> The results in Figure 3 do not match the results in Tables 1 and 2.\n\nWe apologize for the confusion. The results in Figure 3 are achieved with randomly selected demonstrations (both in-domain and out-of-domain). The results in Table 4,5 are achieved with our proposed retrieval methods. In Section 2, we aimed to analyze what aspects of gold in-domain annotations provide the most benefits. As a result, we did not consider complex demonstration retrieval methods. We used randomly selected demonstrations for both in-domain and out-of-domain, and the experiments were repeated 3 times. For \nThe results in Section 5 are about our proposed demonstration selection framework OSID, where we used the proposed SimSQL and CovSQL to retrieve examples.\n\n\n> is the difference due to different demonstration selection methods? Will the finding hold using the same selection method.\n\nThe difference between the results in Figure 3 and in Tables 1 and 2 is due to the chosen method of demonstration selection, as we clarified in response to the earlier question. Regarding the finding in Figure 3, the experiments in Figure 3 are conducted where the selection method remained the same (random selection in three separate runs). It's also worth noting that when comparing \"ID Examples\", \"ID NLQ Distribution\", and \"ID SQL Distribution\", the exact same examples are used across each run. The only distinction is the aspect of the gold in-domain examples that are presented to the LLM.\n\n\n\n\n\n> What’s the number of demonstrations used in each method in Table 1 and does the number of demos affect the results? If so, an analysis on this would be valuable.\n\nIn all our experiments, we consistently maintain the number of examples per out-of-domain database at 5 to reduce the complexity of experiments. For the results in Tables 1 and 2, we configure 4 out-of-domain databases (each with 5 examples) and 5 synthetic in-domain examples for both spider-dev and KaggleDBQA, resulting in the utilization of a total of 25 examples. These numbers are regarded as hyperparameters, which are selected based on the results obtained on a randomly selected subset of 20 databases from the Spider training set. \n\nWe observed that the performance is affected by the amount of demonstrations. When increasing the amount of both in-domain demos and out-of-domain databases, we witnessed a pattern wherein Codex's performance initially ascends and subsequently descends.\nWe also found that the optimal number of examples is tied to the chosen retrieval method. For example, the optimal number of out-of-domain examples is 15 for random retrieval (as shown in Figure 1a) but the optimal amount of out-of-domain examples is 20 for SimSQL. \n\nBeing the pioneering effort to harness a hybrid-source demonstration approach (combining out-of-domain and synthetic in-domain data), our work focuses on demonstrating the effectiveness of the proposed framework. We believe that conducting a comprehensive analysis of the impact of the number of demonstrations is important in the ODIS framework and regard this as future work.\n\n> Minor question: DIN-SQL method shows using GPT-4 outperforms Codex by 7% accuracy. Will applying ODIS to GPT-4 achieve even higher accuracy? Providing a discussion based on a subset is ok if the budget is limited.\n\nWe greatly appreciate this invaluable suggestion. We conducted an experiment with GPT-4 in the final prediction stage on a subset of Spider due to our limited budget. Specifically, we focused on the extra-hard examples, as defined in the Spider paper [1], which are recognized as the most challenging examples in terms of SQL structure. This subset, drawn from the Spider development set, contains a total of 166 examples. We believe that employing the most challenging data affords a clearer illustration of our methodology, within the constraints of a modest budget.\nWe use identical prompts for the studies with Codex and GPT-4, where the demonstrations were selected based on Codex's initial predictions. We suspect that substituting Codex with GPT-4 for the demonstration retrieval stage could yield a further enhancement for ODIS. However, this transition incurs significant costs, hence rendering it unfeasible at the present juncture.\n\nThe table below contains our new experiment with GPT-4 on the Spider extra-hard \n|Method|LLM|Spider extra-hard|\n|---|---|---|\n|ODIS|Codex|65.1|\n|ODIS|GPT-4|68.7|\n\n\n\nMoreover, to demonstrate the effectiveness of our proposed method with other LLMs, we conducted an experiment with ChatGPT (gpt-3.5-turbo-16k), which is more than 10X cheaper than GPT-4, and Code Llama (CodeLlama-34b-Instruct), which achieves the best performance on code-related tasks among open-sourced LLMs, on the entire Spider and KaggleDBQA datasets. The results are presented below. Both the out-of-domain demonstration only method and synthetic in-domain demonstration only method outperform the zero-shot setting on two datasets, with GPT-3.5 and Code Llama. Notably, the ODIS approach shows even more improvements in comparison to both the in-domain only and out-of-domain only methods. Due to our limited budget, we did not undertake a hyperparameter search for the experiments with GPT3.5 and Code Llama. Nevertheless, the current results already illustrate the generalizability of our approach. \n\nAs a result, we believe the remarkable performance of ODIS with both closed-source and open-sourced LLMs on two cross-domain text-to-SQL datasets is able to demonstrate the generalizability of our proposed approach.\n\n\n|Method|LLM|Spider|Kaggle|\n|---|---|---|---|\n|Zero-shot|GPT-3.5|75.7 | 25.7|\n|Out-of-domain Only|GPT-3.5| 80.7| 45.6|\n|Synthetic In-domain Only|GPT-3.5|78.5 | 33.1|\n|ODIS|GPT-3.5|**81.5** |**52.9**|\n||||\n|Zero-shot|Code Llama|70.8 | 18.8|\n|Out-of-domain Only|Code Llama| 76.0| 35.7|\n|Synthetic In-domain Only|Code Llama|78.0 |34.9|\n|ODIS|Code Llama|**78.7** |**40.4**|\n\n[1] Yu, Tao, et al. \"Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe analysis of which part of the demonstration is important is valuable. Although the analysis is only done for the text-to-SQL task, I believe it can be easily extended to other tasks.\n2.\tThe idea of combining synthetic in-domain and real out-of-domain demos during in-context learning is novel and effective, which shows SOTA results for two datasets.\n", "reasons_to_reject": "1.\tThe results in Figure 3 do not match the results in Tables 1 and 2. For example, in Figure 3, out-of-domain only demonstrations show very poor performance (around 78%). However, Table 1 shows 82.1%. And it seems that the proposed method (85.2%) achieves better performance than using the gold in-domain demonstrations (around 84%).", "questions_to_authors": "1.\tFor Weakness 1, is the difference due to different demonstration selection methods? Will the finding hold using the same selection method.\n2.\tWhat’s the number of demonstrations used in each method in Table 1 and does the number of demos affect the results? If so, an analysis on this would be valuable.\n\nMinor question: DIN-SQL method shows using GPT-4 outperforms Codex by 7% accuracy. Will applying ODIS to GPT-4 achieve even higher accuracy? Providing a discussion based on a subset is ok if the budget is limited.", "paper_topic_and_main_contributions": "This paper proposes to use both synthetic in-domain and out-of-domain text-to-SQL data to improve the performance of text-to-SQL task for a new domain for large language models with in-context learning.\n\nThe paper first examines the importance of two parts in the in-domain demonstration: the text (natural language question) part and the SQL part, and obtains the observation that SQL part contributes more during the in-context learning process.\n\nBased on this finding, they propose to use synthetic in-domain data where the SQL queries are first generated and then translated into natural language questions following previous works. Combined with the out-of-domain demonstrations, they propose a demonstration selection framework ODIS. Both the in-domain and out-of-domain demonstrations are selected based on similarity (or coverage) with the input data.\n\nExperiments show both the out-of-domain demonstration only method and synthetic in-domain demonstration only method outperform the zero-shot setting using Codex on two datasets by 5-6% execution accuracy. ODIS further shows 3% execution accuracy over in-(out-of-) domain only method.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for selecting demonstration instances for in-context learning of generating SQL queries from text descriptions with LLMs. Unlike existing approaches, which focus on in-domain in-context learning, which requires hand-annotated in-domain examples, the proposed framework ODIS (1) selects the most useful demonstrations from other domains and (2) generates synthetic in-domain demonstrations. The experimental results reported by the authors demonstrate gains over state-of-the-art approaches on two different cross-domain text-to-SQL benchmarks (although the size of the gains is drastically different between the two datasets). 

While the retrieval of NLQ(uestion)-to-SQL demonstrations from other domains is already an established practice, the authors show, through a series of soundly executed ablation experiments, that it is the combination of retrieval out-of-domain demonstrations and synthesized in-domain examples that yields optimal performance. The gains over zero-shot NLQ-to-SQL have been reported for three LLMs (Codex in the submission, and then additionally ChatGPT and LLama in the rebuttal), indicating that ODIS is a robust approach, agnostic to the backbone LLM. A particularly valuable aspect of this work, which would be informative for future research efforts in NLQ-to-SQL, is the initial analysis of the distribution of SQL constructs in the in-domain demonstrations and the corresponding effectiveness on in-context learning performance. 

While the work is methodologically sound (especially the aforementioned analysis), the extent of novelty of both (1) similarity/coverage-based retrieval of out-of-domain examples as well (2) synthesis of in-domain examples from the NLQs with LLMs, is debatable, and subjective impressions of what is "novel enough" do permeate the reviews. I would, however, hold one thing against the empirical thoroughness of this work: lack of empirical comparison against (a) gold in-domain few-shot in-context learning (as the reviewer TqYu says: what happens if you annotate 10-30 gold in-domain examples) and (b) few-shot fine-tuning (in some parameter-efficient manner, e.g., with QLoRA, with a backbone stronger than the somewhat outdated T5), in addition to only in-context learning. While the authors argue that labeling 10-30 gold examples for 20 or so domains of Spider is "extremely expensive", and that this would not be realistic in the real-world with very many datasets, I'd beg to differ: any developer who wants/expects their database to be extensively queries by non-experts would most likely gladly create more than 30 NLQ-to-SQL examples to facilitate their database/app usage. This is why I'm not entirely sure that zero-shot NLQ-to-SQL with LLMs is the most competitive baseline the authors could have adopted -- if annotating a small number of NLQ-SQL pairs (e.g., 30) surpasses the performance of ODIS, then I'd question that ODIS would be widely used and have a large impact. That said, the analysis part that focuses on identifying in-domain SQLs that are more useful, is unaffected by these concerns. And also, these concerns don't seem to be the issue of this work alone, but the whole body of work on cross-domain transfer for NLQ-to-SQL generation.  

Overall, I find this work to be sound and somewhat novel.
Decision: Accept-Findings