Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
Authors: Linyong Nan
Yilun Zhao
Weijin Zou
Narutatsu Ri
Jaesung Tae
Ellen Zhang
Arman Cohan
Dragomir Radev
Venue: EMNLP 2023 Findings (2023)
Keywords: Text-to-SQL
Few-shot
Zero-shot
In-context Learning
Large Language Models

Abstract: In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions. In this paper, we aim to extend this method to question answering tasks that utilize structured knowledge sources, and improve Text-to-SQL systems by exploring various prompt design strategies for employing LLMs. We conduct a systematic investigation into different demonstration selection methods and optimal instruction formats for prompting LLMs in the Text-to-SQL task. Our approach involves leveraging the syntactic structure of an example's SQL query to retrieve demonstrations, and we demonstrate that pursuing both diversity and similarity in demonstration selection leads to enhanced performance. Furthermore, we show that LLMs benefit from database-related knowledge augmentations. Our most effective strategy outperforms the state-of-the-art system by 2.5 points (Execution Accuracy) and the best fine-tuned system by 5.1 points on the Spider dataset. These results highlight the effectiveness of our approach in adapting LLMs to the Text-to-SQL task, and we present an analysis of the factors contributing to the success of our strategy.

Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
Linyong Nan
Yilun Zhao'
Weijin Zou't
Narutatsu Ri Jaesung Tae'
Ellen Zhang'
Arman Cohan 1.3
Dragomir Radev'
Yale University
"Coumbia University
3 Allen Institute for Al
{linyong.nan nan, yilun. zhao eeyale.@el....o edu
Abstract
systems for main reasons. First, neural mod- els have inherent limitations, including limited working memory that is costly to access during in- ference and long-term memory that is unreliable to read from or write to, making it impractical to have them directly read from large-scale knowl- edge sources. Second, understanding how system decides which information to retrieve and how aggregate is crucial for assessing its reliability and robustness.
In-context learning (ICL) has emerged new approach to various natural language pro- cessing tasks, utilizing large language models (LLMs) to make predictions based context that has been supplemented with a few exam- ples or task-specific instructions. In this paper, we aim extend this method to question swering tasks that utilize structured knowledge sources, and improve Text-to-SQL systems by exploring various prompt design strategies for employing LLMs. We conduct systematic investigation into different demonstration se- lection methods and optimal instruction for- mats for prompting LLMs in the Text-to-SQL. task. Our approach involves leveraging the syn- tactic structure of an example's SQL query to retrieve demonstrations, and we demonstrate that pursuing both diversity and similarity in demonstration selection leads to enhanced per- formance. Furthermore, we show that LLMs benefit from database-related knowledge aug- mentations. Our most effective strategy outper- forms the state-of-the-an system by 2.5 points (Execution Accuracy) and the best fine-tuned system by 5.1 points on the Spider dataset. These results highlight the effectiveness of our approach in adapting L.Mss to the Text-to-SQL. task, and we present an analysis of the factors contributing to the success of our strategy.
Recent investigations have demonstrated the ef- fectiveness of the neural symbolic approach in pro- ducing transparent reasoning process in formal lan- guage sequence (such as Text-to-SQL) for question answering tasks based on databases or knowledge graphs (Berant et al., 2013; Zhong et al., 2017; Yu et al., 2018; Yin and Neubig, 2018; Yu et al., 2019; Ren et al., 2021; Cheng al., 2023). A typical sys- tem comprises a neural semantic parsing module that translates user queries in natural language formal language sequences (e.g., logical forms or executable code) and a symbolic reasoner module, such as database management system (DBMS), that executes the code on structured knowledge sources to extract the result. The primary objective of this work is to improve the semantic parsing module, is essential in extracting answers from relational databases using SQL the formal language.
1 Introduction
Current semantic parsing modules can be broadly categorized based on their learning strate- gies. Sfate-of-the-aar systems involve fine-tuning a pretrained language models on a large corpus of (question, SQL) pairs, enabling the model to generate code (Wang et al., 2020; Yin et al., 2020; Scholak et al., 2021; Xie et al., 2022; Li et al., 2023). Alternatively, the in-context learning (ICL) approach exploits the inherent capabilities of large language models (LLMs) directly produce SQL code by providing a well-defined task prompt (Xie et al., 2022; Chen et al., 2022; Rajkumar et al., 2022: Ni et al., 2023). Existing research indicates
Question answering using structured knowledge source a critical function information retrieval systems that act as an interface between humans and vast structured data repositories. Extracting and aggregating information accurately is funda- mental requirement of these systems and is thus primary goal in their design. In recent years, the neural symbolic design approach (Berant et al.. 2013; Yao and Van Durme, 2014; Liang et al., 2017; Gardner et al., 2018; Yu et al., 2018; Cheng et al., 2023) has become the preferred choice for such

---

that LLMs using prompt-based semantic parsing underperform their fine-tuned counterparts (Liu et al., 2023), while recent studies also suggest that performance of ICL-trained LLMs is significantly affected by the structure of the prompt (Liu et al., 2022; Rubin et al., 2022; Lu et al., 2022; Wei et al., 2022; Fu et al., 2023; Ye et al., 2023). This moti- vates us examine various prompt configurations for semantic parsing tasks, taking advantage of the latest advancements of LLMs pertaining to our do- main of interest.
strations from pool of annotated examples for each test instance is another critical design choice that can impact the ICL performance. We proposed enhancements for each of these components and evaluated them against existing methods.
2.1 Demonstration Selection
The goal is to select subset of annotated examples from pool that offers the best context for solving the test problem. While random selection from the pool is one option, Liu et al. (2022) proposed KNN-augmented example selection (KATE), which retrieves k nearest neighbors from the pool based on the input the compared instances. To achieve this, all the pool instances are first transformed into continuous vectors using a sentence encoder. During inference, the input of test instance is projected into a latent space using the same encoder and then compared the pool of vectors using similarity measure, such negative Euclidean distance or cosine similarity. Finally, the top k most similar annotated examples are selected from the pool.
Our study focused on exploring various prompt design strategies for semantic parsing tasks in the Text-to-SQU domain. We conducted systematic investigation into different demonstration example selection criteria and instruction formats on Text-to- SQL datasets. Specifically, we propose employ an example's s SQL syntactic structure the basis for retrieving demonstrations, thereby facilitating a more accurate representation of the problem struc- ture. Our approach revealed that selecting demon- stration examples with a dual emphasis on diversity and similarity objectives yields maximized gain in performance. Our study also showed that LLMs benefit from latabase-related knowledge augmen- tation in certain circumstances. Through experi- ments, we identified the most effective strategy, which resulted in an Execution Accuracy score of 84.4 on the Spider dataset (Yu et al., 2018). This score is 2.5 points higher than the state-of-the-art system (Ni et al., 2023) and 5.1 points higher than the best fine-tuned system (Scholak et al., 2021) at the time of writing.' These results demonstrate the effectiveness of our in-context learning scheme in adapting LLMs our target task. Furthermore, we present the empirical findings and analysis on the factors that contributed to the success of our strategy.
Structured Prediction as Basis for Retrieval We propose utilizing the output SQL queries to select the demonstration examples, rather than us- ing the input questions. This is because, unlike many tasks where the output is a classification la- bel or extracted entity with little information about the problem structure, Text-to-SQL demands struc- tured prediction which contains more explicit in- formation about the problem structure than that provided in the input question. Furthermore, un- like natural language questions that can only be converted into continuous semantic vectors, SQL queries can be easily transformed into discrete fea- ture vectors based on their syntax, making their comparison more efficient and transparent. To im- plement our proposal, we begin by converting the SQL queries of all pool instances into discrete syn- tax vectors. This is done by parsing the queries and identifying their syntactic elements, including key- words, operators, and identifiers. Each SQL query is then mapped to a "Bag-of Sere-SSttatii--leenne feature vector, each entry of which indicates the presence of a syntactic element in the qucry, i.c., assign 1 (instead of the count) if element is present in the SQL query. During inference. we first generate a draft of the SQL query using a preliminary predictor. We then apply the same process to convert this draft query into a discrete
2 Methods
To design prompts for in-context learning in zero- shot or few-shot settings, it is important to find an optimal way to represent, augment, and arrange all resources in the onput-output mapping. Addition- ally, the task instructions should be formulated to align with these resources. When few-shot learning is employed, the selection of subset of demon-
'Our comparison focuses on fine-tuning studies that em- ployed standard Transformer architecture without any layer modifications or the inclusion additional modules.
WWo will open source our code for experiments: https: //anonymous. url

---

vector, which is used to represent the test instance for retrieving demonstration examples.
knowledge sources such as databases tables have been linearized into a "text" sequence. Building on previous methods (Rajkumar et al., 2022), we adopt a representation of the database using a "code" se- quence, specifically the CREATE query employed to construct the table initially, as illustrated in listing and 2 of the Appendix. This linearization approach provides data type information for each column and encompasses all foreign key constraint details within the database. Moreover, we modify other resourees in the instructions, such as the question and example entries the database, to conform to the code sequence style by appending them as comments.
Balancing Diversity and Similarity We propose a new demonstration selection strategy that differs from Liu et al. (2022), which retrieves the most similar examples with continuous-valuee measure- ments for each test instance. In contrast, our strat- egy seeks to balance similarity and diversity of the demonstrations. This is achieved by chang- ing the representation of the given example from a continuous-valun vector denoting the question semantics to a discrete- valued vector that captures the syntax. To obtain demonstration examples that are similar to the given example, we first split the pool of annotated examples into disjoint par- titions that represent different categories. Specifi- cally, we use the difficulty-Ievel based categoriza- tion derived from the Spider dataset (Yu et al.. 2018), because it is developed strictly based on syntactic coverage and structure of SQL query, ensuring that queries satisfying the same condi- tions are grouped into the same category. While alternative categorization options may exist, we leave this for exploration in future work. Given a test instance, we use preliminary predictor to generate draft SQL query and, based on its cat- egory, retrieve candidate examples that belong to the relevant partition. Next, to select diverse exam- ples from the candidate partitions, we implement k-means clustering on the discrete vectors of exam- ples, selecting & diverse examples that are closest to each centroid of the cluster. The resulting ex- amples exhibit similarity to the test example by sharing the same category, yet maintain diversity in problem structures. These demonstrations are then used to construct the prompt. The procedure for our demonstration selection strategy is outlined in Algorithm 1 of the appendix.
Schema-related Knowledge Augmentation The ontology of a database delineates the structure and semantics of the database by offering defini- tions for a set of classes (tables), their attributes (columns), and the relationships among them. We initially enhance the semantics of each class and attribute by claborating on their meanings within the context of the entire database. Specifically, employ OpenAl's gpt-3. 1..33.5tuu 5-turbo engine? to generate natural language definition for each column in every table, considering all its values and other columns. We then incorporate these definitions into the input either by appending them as a block comment or inserting them within the CREATE query as inline comments. Furthermore, we suggest augmenting the repre- sentation of the database structure by providing an Reaiit--Relaiinshhip summary that outlines the connections between tables and specifies how they can be joined. As depicted in Figure 9 of the Appendix, an Eelaty-Relationship diagram of a database utilized to enumerate all possible paths between distinct tables. These paths are subsequently arranged in descending order based on their respective lengths. The resulting summary has shown to be useful in our experiments for test instances where multiple tables need to be combined. Listing 5 further demonstrates our augmentations and how we arrange them to construct the prompt.
2.2 Schema Representation in Instruction
Instructions are crucial to designing prompts, as they define the task by clarifying how provided re- sources can aid the inference process (Dong et al., 2023). Our primary focus lies in determining the optimal way to represent structured knowledge source within the instruction and identifying sup- plementary resources that can enhance the infer- ence process.
2.3 Integrated Strategy for Text-to-SQL
Upon examination, we found that models trained with ICL exhibit sensitivity to the number of demonstration examples, resulting in noticeable variance in performance across models provided
Linearization of Structured Knowledge We be- gin by altering the lincarization of structured knowl- edge. In prior research (Xie et al., 2022), structured
Public API available at https:/ https://ppeeaii.cmm/pp///

---

with various numbers of demonstrations. To estab- lish substantial conclusions when comparing dis- tinct prompting approaches, we present the mean and standard deviation for models sharing identi- cal configurations except for the varying number of demonstrations. In addition, we employ a ma- jority vote on these models exhibiting diverse per- formances. Specifically, we obtain the execution results of different models' greedy decoding predic- tions, eliminate those with execution errors by de- terministic database management system (DBMS), and choose the prediction that receives the major- ity vote. Alternative integration methods, such as the self-consistency sampling (Wang et al., 2023), are also available, but we reserve their exploration for future research. The comprehensive results are available in Figures 10, 11, 12 of the Appendix for reader's s perusal.
Spider-Syn (Gan et al., 2021a) replaced schema-related words in the questions of Spi- der examples with manually selected syn- onyms that reflect real-world question para- phrases to evaluate the robustness of systems.
Spider-DK (Gan et al., 2021b) defined five types of domain knowledge and modified some Spider examples by adding domain knowledge to evaluate the cross-domain gen- eralization capability of a given system.
Spider-Realistic (Deng et al., 2021) removed explicit mentions of column names from Spi- der examples to reflect more realistic text- table alignment settings, and selected eight ex- isting Text-to-SQU datasets for cross-domain evaluation.
We propose the following procedure for con- structing prompts for the Text-to-SQL task. Given a set A of annotated examples, we first establish a categorization that divides the pool into disjoint partitions A A ...... with each partition contain- ing examples whose SQL queries share relatively similar syntax structure. Next, we apply the k- Means strategy detailed in Section 2.1 obtain diverse demonstration examples D for partition A For each example, the demonstration is con- structed by transforming the database into multi- plc CREATE queries and augmenting with schema- related knowledge. During inference, we employ preliminary model to generate draft SQL query, which is used to determine the problem category and thus the corresponding D for building the prompt. We obtain multiple predictions using vari- ous numbers shots in D and perform majority voting to arrive at the final prediction. Details of this approach are shown in Algorithm 2 of the ap- pendix.
Model We evaluate different ICL strategies with Codex (Chen et al., 2021), a GPT-3 variant that was finetuned on code data on the web and has demon- strated ofte--of-thh-art performance as the time of writing (Ni et al., 2023). Specifically, code dode-ddvinci--e -002 engine and present the results of systems with prompts ranging from I to 10-shot. Additionally, we report the few-shot results utiliz- ing the ChatGPT (gpt-3. 3.t--3.5ttte 5-turbo) model. How- ever, due to its maximum context length limitation of 4096, we only obtain results for systems pro- vided with prompts ranging from I to S-shot.*
use the
Evaluation Metric We use execution accuracy as the evaluation metric for all experiments, which measures the percentage of system predictions lead- ing to the gold execution result.
We compare the following prompting strategies for generating SQL queries in few-shot and zero-shot settings.
Baselines
Few-shot
3 Experiments
Random sampling (R): Select demonstration ex-
3.1 Experimental Settings
amples randomly from the pool.
Similarity sampling (S)
Dataset We conduct comprehensive experiments on the following four semantic parsing datasets:
Diversity sampling (D): Select diverse examples from k-Means clusters of the pool.
Similarity Sinilarity-iverriity sampling (SD): Select ex-
Spider (Yu et al., 2018) is a cross-domain se- mantic parsing dataset that contains complex Text-to-SQL problems. The data originates from 200 databases covering 138 different do- mains. We use the 7,000 training data as our pool of annotated examples.
amples based on Algorithm 1.
SD + schema augmentation (SA): Enhance in- structions with schema knowledge (semantic aug- mentation or structure augmentation).
PPblic API available at https://openai..omm/np

---

Execurtion Accuracy
5dder
splderaym
Spide-UK
Random Sampling
Diversity Sampling
SD-Scheme Augmentation (SD+SA)
Similarity Sempling
Similarity Diversity sampling (SD)
SD-SA+Voting (integrated]
(a) Few-shot results
Execution 20
Solder
Spicen Syn
Quleeen
Dataseets
Baselne
DB as IMct-NG
Semantic auqmentation ads colume summary (nine comment)
Strue are aucmentation and onteroey summary Structure sugmentaiion and ontorogy summars tpratich sllesing)
Daseline: D8 25 code
Semantic sugmentation ads colume summary (lbck commenti
(b) Zero-shot results
few-shot setting, error bars indicate means and standard deviations performances of systems provided with prompts ranging from 4-shot to 10-shot. To obtain the етror bars for the random sampling approach, we conducted independent runs using different random seeds. Schema augmentation utilized the reported results in (a) is structure augmentation add ontology summary. In the zero-shot setting, the eтror bars indicate means and standard deviations over 3 independent runs. Our results suggest that 1) using similarity and diversity objectives in the sampling process, 2) including schema representation instructions, and employing model voting with different shot outcomes both contribute the improvement of ICL performance.
Figure Few-shot zero-shot results Codex
datasets.
SD SSA Voting: Integrated strategy described in Algorithm 2.
efficacy across multiple datasets. The evaluation of demonstration sampling strategies in a few- shot setting testing on code-davinci-on is illus- trated Figure 1a, and more few-shot results 3pt-3.5-turbo 5-turbo are shown in Figure 2. We com- pared different demonstration selection strategies, including random selection, k-nearest neighbors selection (similarity sampling) k-means selee- tion (diversity sampling), and our proposed ap- proach, which combines both similarity and di- versity. Moreover, we examined the impact of augmenting schema representation within the task instructions and assessed the performance of our
Zero-shot
Baseline DB as text-seq: Standard prompt for Text-to-SQL task, where structured knowledge is linearized as sequence.
Baseline DB code-seq: Improve instructions by linearizing structured knowledge source as multiple SQL CREATE queries.
Baseline DB as code-seq + SA: Enhance in- structions with schema knowledge.
3.2
Main Results
'Due to the deprecation of the Codex API in March 2023, similarity sampling experiments were only conducted on the Spider datasct.
this section, we present a comprehensive analy- sis of various prompting strategies, assessing their

---

integrated strategy, Our findings indicate that em- ploying similarity and diversity objectives in the sampling process leads to better performance on average across all datasets. Furthermore, incorpo- rating schema representation within the instructions enhances performance, and the implementation of voting of models with different shot results in marked improvement in overall performance.
demonstrations in order to effectively understand and utilize the provided information. In future study, we will explore better structural schema aug- mentation that aligns to the zero-shot setting.
4 Analysis
4.1
Prediction-Syntax based Retrieval
The existing method for selecting demonstrations relies on the semantic representations of the ques- tion and the database. We propose an alterna- tive method specifically for code generation tasks, which focuses on the syntax of the solution code. We examined syntax coverage and syntax similarity of the prompts produced with different strategies. Syntax coverage computed by counting the occur- rence of syntactic elements (keywords, operators. and identifiers), and dividing it by the total number of all syntactic elements. Syntax similarity, on the other hand, measured by the mean Euclidean distance between the discrete vector representation of the predicted SQL and vectors that represent the gold SQLs of the demonstrations selected. As indicated in Table 1 of the appendix, both of these metrics contribute the quality of the examples selected. Furthermore, simple summation of the two measurements suggests a correlation with the system's 's performance, as illustrated in Figure of the appendix. We argue the efficacy of our strat- egy through the following rationale: (1) in cases where the pool of annotated examples is limited in diversity of the problem structures, certain test problems may lack similar examples available for retrieval; and (2) neither the semantic represen- tation of the question/databass nor the distance metric inherently support encapsulation and com- parison of different problem structures, whereas SQL syntax provides direct measurement of the problem structures. Given these constraints, the optimal strategy is to select similar examples while ensuring the coverage of many syntax demon- strations feasible to mitigate potential failures in similarity-based retrieval.
Execution
72
Sampling
Similarity- sampling (5i)
Similority Samppnng
5Ds Senems Augiennattinn (50- SAA
Eiversily Samcling
5D SA Virting Unviage sbud)
Figure 2: Few-shot results of gpt-3. 5-turbo for Spi- der. Error bars indicate means and standard deviations over performances of systems provided with I-shot to 5-shot prompts. Schema augmentation utilized for the reported results is semantic augmentation add column summary as block-comment.
The efficacy of schema augmentation is fur- ther supported by experiments in zero-shot set- ting, as illustrated in Figure 1b. We compared systems using different linearization methods for prompts: one that transforms the database into a text sequence, and another that uses multiple CREATE queries to represent the database. The latter method shows noticeable improvement in performance. We also contrasted two separate techniques for augmenting schema representation: one that adds semantic information to each col- umn within each table, and another that incorpo- rates entity-rlaaionssip knowledge into the schema. The results suggest that structural augmentation (adding ontology summary) brings slight greater improvement in the few-shot setting for Codex (shown in Figure 5), while semantic augmenta- tion (adding column summary as block comments) proves more beneficial in the zero-shot setting for Codex and also the few-shot setting for ChatGPT (gpt-3.5-turbo) 5-turbo). We hypothesize that this dif- ference may arise from the less descriptive nature of structural augmentation, which calls for more
4.2 Comparative Analysis of Retrieval Methods
We conducted examination of various similarity- based retrieval methods and presented compar- ative analysis of their performance in Figure 3. The primary variable in this investigation was the representation extracted for each example, with a focus on extracting and comparing the following

---

79
79
79
Execution
77
77
77
76
75
Syntax
all L6-2
3.2011e-bwee121 CooNTT-Aewe
fentural
liwey Tastares
elagario Lao vatuano
RoBESTu-bazz
scceacs--oaaajj Hurajgn codeet -taeee)))
noat-embedding-ee sawpddddii-g---002
entesonng 000/0-D0 c0d4-000
cresesorng
Figure 3: Comparison between various similarity based demonstration selection methods, Q indicates the embedding model employed to extract representation for the question; stands for database, and stands for SQL query.
CodeTT-base (Wang et al., 2021) CodeBERT- base (Feng al., 2020) for encoding databases; (3) syntactie embeddings of predicted SQL, generated by either binary coding indicate the presence of SQL syntactic elements or by quantifying their occurrences; and finally, (4) embeddings that en- code questions, databases and predicted SQL using mexxtembedding-ada
Exocution Accuracy
7B
24
70
Methods
The following conclusions can be drawn about the similarity-based retrieval methods for Text-to-SQL task: (1) questions alone ef- fectively represent distinct examples for re- trieval purposes; (2) RoBERTa-base provides better embeddings for comparisons relative to www.-eebbddinng-ddd--aaaaa
brarp-iyntae wwe.enoyotaudeeeeeooooo
Figure 4: Comparison between various diversity based demonstration selection methods.
(3) it is feasible employ models that have not been fine-tuned on Text-to-QQL exammll for similarity-based re- trieval, while still achieving comparable perfor- mance to fine-tuned models; (4) the linearization of databases as SQL queries facilitates the extraction of enhanced embeddings.
Excuton
73
75
choos 10 Choos
ZEro STOt Preepetieett
Kealin
Figure 5: Comparison between various schema augmen- tations in few-shot and zero-shot settings.
Additionally, conducted comparison be- tween multiple embeddings utilized for diversity- based demonstration selection, encompassing em- beddings that encode the semantics of questions, databases and predicted SQL, as well as embed- dings that capture the syntactic features of pre- dicted SQL. As depicted in Figure 4, the syntactic embeddings of predicted SQL serve the most effective basis for contrasting different examples for diversity-based retrieval purposes.
embedding types: (1) question embeddings gener- ated by Sentence-BERT (Reimers and Gurevych, 2019)6 RoBERTa-base (Liu al., 2020), and Ope- nAI's textt-embedding--ad--000 (2) combined question and database embeddings obtained by (i) employing single model (i.e., T5-base (Raf- fel et al., 2020) finetuned on the Spider train- with the database linearized text sequence or CREATE queries, and (ii) utilizing separate models, specif- ically RoBERTa-base for encoding questions and
ing split and sentt-mbeddingg-dd--aaa
4.3 Schema Augmentation
Figure 5 presents the outcomes of various schema augmentations applied the instruction. It is ob- served that improvement not apparent in the few- shot setting; however, in the zero-shot setting, the
'HuggingFace model name: all all-MiniLN-L6-12

---

semantic augmentation incorporating descriptions of all table columns proves to be beneficial.
aspect of enhancing model performance. Several studies (Sorensen et al.. 2022: Gonen al., 2022; Wu et al., 2022; Hu et al., 2022: Lu et al., 2022) have proposed metrics to measure the suitability of examples with respect to the target objective and to determine the optimal ordering of them. Liu et al. (2022) suggest selecting examples that are seman- tically similar to the test example by employing k-NN approach in the embedding space. Rubin et al. (2022) train a prompt retriever based on con- trastive learning, wherein examples are classified either positive or negative if they are ranked among the top-k or bottom-k probabilities of a language model generating the target output, conditioned on the retrieved example and the input. Zhang et al. (2022) suggests to actively select demonstrations using Q-Learning Su et al. (2023) introduces the Vote-k approach to selectively annotate diverse and representative examples for pool construction, then retrieve based on the similarity. In contrast, our approach retrieve a diverse set of examples given a pre-established pool. As the authors demonstrate that having a diverse and representative pool is im- portant for the success of ICL., we posit that a sim- ilar characteristic is equally important when com- posing the prompt, as this approach increases the likelihood of including various syntactical usages or similar problem structures within the prompt.
4.4 Effectiveness Analysis
In order to determine the problem types that ben- efit most or least from our proposed methods, we also evaluate the performance of different mod- cls across various problem categories within the Spider dataset. As indicated in Figure 7 of the appendix, our -diverstttyd strategy proves beneficial for most problem types, with the excep- tion of the medium split, which includes the most diverse problems. This is the case where similarity- based retrieval fails and syntax coverage becomes more crucial. Furthermore, we observe that aug- menting schema semanties more effective for the easy and medium splits (albeit with high vari- ance), while augmenting schema structure is more effective for more complex problems. This ob- vervation leads to hypothesize that challenging problems necessitate addressing a higher number of tables, thus requiring a more comprehensive un- derstanding of the entire database structure. Lastly, the integrated approach is effective across all ex- amples, offering increased benefits especially for those difficult problems.
4.5 Preliminary Models
To assess the impact of the choice of preliminary model used to generate the draft SQL on our ap- proach, we conducted tests involving our methods for preliminary models with varying performance levels. Figure 8 of the appendix reveals that the preliminary models have a relatively minor effect on the performance of the similarity-diversiyy or integrated approaches, exhibiting gradual improve- ments as better preliminary models are utilized.
5.2 Prompt Formatting
Prompt engineering is concerned with investigat- ing the impact of prompt structure on downstream task performance. For tasks that involve multi-step reasoning and higher complexity, Chain-of-thought prompting has been developed (Wei et al., 2023; Kojima et al., 2023). This approach involves lay- ing out the generation process over multiple steps and using the model's own intermediate process as input. Wang et al. (2023) proposes to sample mul- tiple different chain-of-thouuhts then selects the most consistent answer through marginalization of all possible reasoning paths. Press et al. (2023) suggests that prompting LLMs to ask follow-up questions is an effective way to construct the chain- of-thoughts process. Zhou al. (2023) proposes an automatic approach identify the optimal prompt by searching over a pool of model generated in- structions, assigning scores to them, and selecting the prompt with the highest score.
5 Related Work
Existing literature indicates the ability of large lan- guage models to adapt to new tasks at inference time by learning from few example demonstra- tions (Brown al., 2020; Radford al., 2019). This new capability has been referred as in- contexi learning. In this paper, we expand on pre- vious works that investigate the optimal representa- tions for prompt inputs.
5.1 Prompt Organization
Prompt organization investigates the task of select- ing and organizing in-context examples, a critical

---

6 Conclusions
Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates. Inc.
In this study, we investigated various prompt de- sign approaches for semantic parsing tasks in the Text-to-SQL domain. We proposed approach that leverages an example's SQL syntactic structure for demonstration examples selection, emphasising both diversity and similarity as the sampling ob- jectives. Additionally, We found that LLMs gain benefits from database-related knowledge augmen- tations. Future research can build upon our findings to examine the transferability of our approach to other domains. Through ongoing improvement of LLMs' capabilities in semantic parsing, we aim to contribute to the development of QA systems that are more accurate, robust and comprehensible.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained code.
Limitations
One of the main limitations of this study is the re- producibility problem. The experiments presented in this paper relied on the use of OpenAl APTs, which were available at the time of our research but have since been or will be deprecated. This means that the results of our experiments cannot be replicated using the same APIs, which hinders the reproducibility of our findings. To address this limitation, will focus on providing experiments results that are based on open-sourced LLMs (Tou- vron et al., 2023; Taori et al., 2023; Chiang et al., 2023) for greater transparency and reproducibility. Another limitation is that it is not clear how our approach will benefit LLMs given smaller or more constrained pools of annotated examples. Although we postulate that our approach offers the advantage of providing prompt with maximal coverage of similar problem structures when identically struc- tured problems cannot be found in the pool, we could not substantiate this due to our limited bud- get and access to the OpenAI APIs.
Wenhu Chen, Xueguang Ma. Xinyi Wang. and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:221..12588.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages. In The Eleventh Inter- national Conference on Learning Representations.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chathot impressing gpt-4 with 90%* chatgpt quality.
References
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Linguis- tics.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure- Structure-grounded pretraining for text-to-SQU SQL. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1337-1350. Online. As- sociation for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Mors,rt-Voss,,
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning.

---

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Computational Linguistics.
(Volume 1: Long Papers), pages 23-33, Vancouver, Canada. Association for Computational Linguistics.
Aiwei Liu, Xuming Hu, Lije Wen, and Philipp S. Yu. 2023. A comprehensive evaluation of chatgpt's zero- shot text-to-sql capability.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online, Associa- tion for Computational Linguistics.
Yao Fu, Hao Peng. Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-bast -baed prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations.
Yujian Gan, Xinyum Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Peng- sheng Huang. 202 la. Towards robustness of text- to-SQL models against synonym substitution. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 2505- 2515, Online. Association for Computational Lin- guistics.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Rolbert]a: robustly optimized [bert) pretraining approach.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Longg Papers), pages 8086 8096- Dublin, Ireland. Association for Compu- tational Linguistics.
Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b. Exploring underexplored limitations of cross-domain text-to-SLL generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8926-8938 8931, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tau Yih, Sida I. Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code genera- tion with execution.
Matt Gardner, Pradeep Dasigi, Srinivasan Tyer, Alane Suhr, and Luke Zettlemoyer. 2018. Neural semantic parsing. In P Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tu- torial Abstracts. pages 17-18, Melbourne, Australia. Association for Computational Linguisties.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Mcasuring and narrowing the compositionality gap in language models.
Hila Gonen, Srini lyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. In Findings of the Association for Computational Abu
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Mochine Learning Research, 21(140):1-67.
Linguistics: EMNLP 2022, pages Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabilities of large language models.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In AAAI.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks, In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing TEMNLP-IJCNIL pages 3982-3992, Hong Kong. China, Association for Com- putational Linguistics.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. For- bus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics

---

Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michi- hiro Yasunaga, Haitian Sun, Dale Schuurmans, Jure Leskovee, and Denny Zhou. 2021. Lego: Latent execution-guides reasoning for multi-hop question answering on knowledge graphs. In Proceedings of the 38th International Conference on Machine Learn- ing, volume 139 of Proceedings of Machine Learning Research, pages 8959-899999 -9700. PMLR.
and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Representations.
Yue Wang. Weishi Wang. Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 2655-2671, Seattle, United States. Association for Computational Linguisties.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning large language models. In Advances in Neural Information Processing Systems.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau, 2021, PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong. 2022. Self-adaptive in-context learning.
Taylor Sorensen, Joshua Robinson, Christopher Ryt- ting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An cnermatiio-theoreet. approach to prompt enginecring without ground truth labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pa- pers), pages 819-862, Dublin, Ireland. Association for Computational Linguistics.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengeheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang. Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Longuage Processing, pages 602-631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Selective annotation makes language models better few-shot learners. In The Eleventh International Con- ference on Learning Representations.
Xuchen Yao and Benjamin Van Durme. 2014. Infor- mation extraction over structured data: Question an- swering with Freebase. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 956-966, Baltimore, Maryland. Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction followwwn Ilama model. https:// gith wwwm/tats-u/aaeeeeoooooooeaae
laattss-laassae lawpllabsstttae http:/staffrrd..o prd_aaa
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anna Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang. Hyeongu Yun, Yireun Kim, and Minjoon Seo. 2023. In-context instruction learning.
Pengcheng Yin and Graham Neubig. 2018 TRANX:
A transition- baasestiio---edddddd neural abstract syntax parser for semantic parsing and code generation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 7-12, Brussels, Belgium. Association for Computational Linguistics
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozoy, and Mathew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578, Online. Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceed- ings of the 58th Annual Meeting of the Association
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery,

---

for Computational Linguistics, pages 8413 8426, On-
line. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi. Zihan Li, Youxuam Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin- cent Zhang, Caiming Xiong, Richard Socher. Walter Lasecki, and Dragomir Radev. 2019. CoSQL: A conversational text-to-SQL challenge towards cross- domain natural language interfaces databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNE) pages 1962- 1979, Hong Kong, China. Association for Computa- tional Linguistics.
Tao Yu. Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: large-scale buman-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQI. task. In Proceedings of the 2018 Conference on Empirical Methods Natural Lan- guage Processing, pages 3911-3921, Brussels, Bel- gium. Association for Computational Linguisties.
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Ac- tive example selection for in-context learning. Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9134- 9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql:
Generating structured queries from natural language using reinforcement learning. CoRR, abs/ 709.00103.
Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference Learning Representations.

---

Appendix
Algorithm 1: Somillarit--iverraa Demonstration Selection
Input: Set of annotated examples A, test examples T, # demonstrations k, categorization
fa, .....
Result: Set of prompts P. where P. is the prompt for test example Tç
* Split A into disjoint partitions Aa.3...
*/
for A; in annotated ser A do
get_category(A.300)))
A .append(A.);
Vi get_syytaxx.vectors/A....]]
end
* Prepare demonstrations D for each partition A
x/
for partition A in Ao.B.... do
M Ik-Meass_ cetheaansccluteeringt....k
/* Vi is of discrete vectors for examples in AV, M has kt centroids
Pr.... 14
for μr do
D pp
end
end
* Build test prompts
for
T; test set do
T.SQQL - (nta-iiitial_predcctt(((()))
G get_categgoy((..S0L)
P. = build_prompt(D",.T))
end
return P

---

Algorithm Integrated Strategy
Input: Set of annotated examples A, test examples T, # demonstrations k. categorization
{a, ...).. and from Algorithm I: disjoint partitions (-A%, AA ....) and corresponding demonstrations (Da, De,....)
Result: Set of SQL predictions SP, where SP; is the final prediction for example T;
for T; test set do
T.SQQL intttaal.reeeecctr((((
G = get_ get_category(7T5500)
for π 4 to k do
P build_prompt n), T:);
P. augment_schema (P:);; (P:))
SP Model(P!")) );
ERT DBMS(SP?);
end Bewpree_xxe__Errrrseeee
ER
SP, Majority_.otee(R:))
end
return SP
Coverage
Similarity
Execution Aceuracy
Random
0.38
0.24
76.03
Similarity
0.35
0.30
78.33
Diversity
0.43
0.23
78.64
Similarity-Diversityy
0.50
0.26
80.32
Table I: Average syntax coverage and similarity measures prompt for different demonstration selection strategies and the corresponding execution accuracies.

---

Similaritt-Oiversity Sampling,
Execution Accuracy
Diversity Sampling
Similarity Sampling
76
Random Sampling
0.62
0.64
0.66
0.68
0.70
0.72
0.74
0.76
Syntax Coverage Syntax Similarity
Figure 6: Correlation between syntax coverage and similarity measures prompts and execution accuracy.
Execution Accurac)
94
Execution Accuracy
84
82
Easy
Medium
Execution Accuracy
Execution Accuracy
60
86
50
Hard
Extra
sampingg
Similiitt dlverohy
Schera semanbec sugmentatten recormmento
Figure 7: Efffects of various prompting strategies on Text-to-SQL problems of different difficulty levels.
86
82
80
Execution
Redlminary
Itterrte
Smiority Diversity
Experiments
Figure 8: Effects of preliminary model on proposed strategies.

---

I Given the following database schema:
2 gymnast symnast_id, #lor_exerrise_points pomeel_horse_points rings_points vault_points, parallel_bars_points people people_id, name, age, height, hometown
corizontal_bar_poinee points
Answer the following: Return the total points of the gymnast with the lowest age.
select t1. total_points from gymnast as t1 join people as t2 on t1. gymnast_id
t2. people_id id order by t2. age ase limit 1
Listing 1: Baseline prompt with text representation of the database.
7* Given the following database schema: */
CREATE TABLE IF NOT EXISTS "symnast" (
"Gymnast_II0" int
FFooor_Exercisc_ooinss real,
"Pommel_Horse__ooie real,
"Rings_Points" real.
"Vault_Points" real.
"Parallel_Barr_Poontt" real, "Horicontal_BarPPiinss"" real, "Total_Points" real, PRIMARY KEY ("Gymnast_ID"), FOREIGN KEY ("Gymnast_ID") REFERENCES "people" "aropl""("Peppp""" ("Peaple_ID") ID") );
CREATE TABLE IF NOT EXISTS "people" "People_ID" int, "Name" " text,
"Age" real,
"Height" real,
"Hometown" text,
PRIMARY KEY ("Peaple_ID")
);
22
23 /* Answer the following: Return the total points of the gymnast with the lowest age. */
24
25
select t1. totatoollpppp points from gymnast as t1 join people as t2 on t1. gymnast_id id
t2. people_id order by age ase limit
Listing 2: Baseline prompt with code representation of the database.

---

1. /x Given the following database schema: x/
CREATE TABLE IF NOT EXISTS "department" (
"Department_ID" int, -- a unique identifier for depar
"Name" text, -- the name of the department "Creation" text,
the date the department was created
"Ranking" int, -- the ranking of the department within the organization "Budget_in_Billi" _n_Biiioons" real, the department 's budget in billions of dollars
"Num_Emp Empeessss" real, the number of employees in the department PRIMARY KEY ("Department_IDD)
);
CREATE TABLE IF NOT EXISTS "head" (
"head_ID" int, -- unique identifier for the head of department "name" text, -- the name of the head of the department "born_state" text, -- the state where the head of the department born "age" real, -- the
of the head of the department PRIMARY KEY ("head_ID")
):
CREATE TABLE IF NOT EXISTS "management"
" "department_ID" int, the unique identifier for the depar tment being managed
"head_ID" int, the unique identifier for the head of the department "temporary.acting" text, whether the head of the department serving temporary or acting capacity PRIMARY KEY ("Department__D" "head_ID")
FOREIGN KEY ("Department_ID" REFERENCES "department ("Department_I0") FOREIGN KEY ("head_ID") REFERENCES "head" ("head_ID") ):
/a Answer the following: What are the distinct creation years the departments managed by a secretary born state 'Alabama '? */
79 select distinct t1, creation from department as e1 join management as t2 on t1, department_jo t2, department_id join head as on t2. t3. head_id where t3. born_state "Alabama'
Listing 3: Prompt with semantic augmentation of the schema as inline comment.

---

1. /x Given the following database schema: */
2
CREATE TABLE IF NOT EXISTS "department" (
"Department_ID" int, "Name" text,
"Creation" text,
"Ranking" int,
"Budget_ in Billiini real,
"Num_Employees" real,
PRIMARY KEY ("Department_ID")
);
CREATE TABLE IF NOT EXISTS "head" (
"head_ID" int, "name" text, "born_state" text, "age" real, PRIMARY KEY ("head_ID")
):
CREATE TABLE IF NOT EXISTS "management" "d "department_ID" int,
"head_ID" int, deeporaar_actting text, PRIMARY KEY ("Department_IID", "head_ID"))
FOREIGN KEY ("Department_ REFERENCES "department (aaprrtmeennn"DDaaaaa""""" ("Department__DD" FOREIGN KEY ("head_ID") REFERENCES "head" " ("head_ID")
);
/x Table column descriptions:
('department ('Department_ID'; 'a unique identifier for a department's 'Name :: 'the name the department's 'Creation's 'the date the department was created 'Ranking's 'the ranking Budget_ii_Biillonss : "the department budget in billions dollars", "lum_Emplyyess the number of employees in the department ), 'head': {' 'name 'the 'the state where the
the department within the organization
head_ID ' unique identifier for the head of a department name of the head of the department 'bornsstat''s head of the department was born', "age" ; 'the age of the head of the department )), "management's d'department_DD" the depar tmen't being managed', "head_ID's "the unique identifier for the head of the department', "temporary_acting department is serving in a temporary or acting capacity '}} 2/
"the unique identifier for
'whether the head of the
/x Answer the following: What are the distinct creation years of the departments managed by a secretary born in state 'Alabama'? '? */
select distinct creation from department as t1 join management as t2 on t1. department_iis = t2. department_id join head as t3 on t2. head_id = t3 head_id where t3. born_state "llabama"
Listing 4: Prompt with semantic augmentation of the schema as block comment.

---

INTOOOR
contieres
TDIT
Contee
INTEGER
Counryyd
ENTOGOR
T00
RCAL
Country
TEIT
Weight
INTOGOR
continents. contid -> countries. continent, couttries.continent, car_eakers country, car_ hor_makerss.tt 1d model, naker, nodel_list, mide model, makeid cars_data.id
countries, countryid ->
INT0GO0
CLAMK 2008
COURI
YUM
CLAG S4c1800
HBoT HIGH D0CREE
vendwek
DEGOLL
DNROUL SPACE
DEPT.002 2002
enployee. enp emp..nnr nun deperteent. erp_nus, department, dept.. -> course, course,dept_com dept_code,
course, cerrssecca__ue cra_code >> $lass.cra_coode code, class. clwss,ascccccoecce _lass_code anroil.class_com code depart departnent.deg dept.
-> student.dept.comm enployee, enp_.nin class.prod _rou_nnn booyyeeeen__nui
student.stu.nun >> enroll.stu.nn stu
enployes.
professor. emp_nun
degartment.dee
professor depeessordeeptcooo cade
Figure Examples of schema structure representation construction.

---

1. /x Given the following database schema: */
CREATE TABLE IF NOT EXISTS "continents" (
"ContId" INTEGER PRIMARY KEY, "Continent" TEXT
);
CREATE TABLE IF NOT EXISTS "countries" (
"CountryId" INTEGER PRIMARY KEY,
"Coun tryName" TEXT, "Continent" INTEGER, FOREIGN KEY (Continent) REFERENCES continents continents(Contion (ContId)
):
CREATE TABLE IF NOT EXISTS "car_makers"
"Id" INTEGER PRIMARY KEY,
"Maker" TEXT,
"FullName" TEXT,
"Country" TEXT,
FOREIGN KEY (Country) REFERENCES countries(ouuntryto))
):
CREATE TABLE IF NOT EXISTS "model_ listt
"ModelId" INTEGER PRIMARY KEY, "Maker" INTEGER, "Model" TEXT UNIQUE,
FOREIGN KEY (Maker) REFERENCES car_makers (Id)
):
CREATE TABLE NOT EXISTS "car_names" "MakeId" INTEGER PRIMARY KEY, "Model" TEXT,
"Make" TEXT,
FOREIGN KEY (Model) REFERENCES model_list (Model)
):
CREATE TABLE IF NOT EXISTS "cars_data" (
"Id" INTEGER PRIMARY KEY,
"MPG" TEXT,
35
"Cylinders" INTEGER,
26
"Edispl" REAL, "Horsepower" TEXT, "Neight" INTEGER, "Accelerate" REAL, "Year" INTEGER,
37
40
FOREIGN KEY (Id) REFERENCES car_names (MakeId)
)
/*
Database ontology:
continents. contid -> countries. continent, countries countryidd -> car car_makers. country, car_makers.. id -> model_list maker, model_list.mode -> car_names. model, car_names. makeid -> cars_data.
*/
/a Answer the following: How many continents are there? x/
select count (*) from continents;
Listing 5: Prompt with structure augmentation of the schema.

---

der
Spider-yy
Execunion Accurscy
22
Spider
Realistic
20
Number demonstration
Sivwerkyy.wwrrstt samping
Diversity samping
Smilaryyttandde sampling
Randem sampling
+
Simfarity samaling
Figure 10: Few-shot results for comparing different sampling strategies with different number of demonstra- tion examples selected for the prompt.
3p der
Spider Syn
01
15
15
Diccuton Accuraz)
Spider
Spider Reslesic
Number demonstrarion examples in the
Instruction
samppng samping ing
sampling schoma semancics sugmentation. commant
strucaure sugmentation. protib
50 samoling schema semaniies sugmenation -hork comment
Figure 11: Few-shot results for comparing different schema representation augmentation methods with different number of demonstration examples selected for the prompt.

---

Spider Easy
Medium
Diocucon Acourscy
Spider-Mard
Spiden-Estrae Extre--stt--arrd
10.0
375
33.0
413
Number damorstration examples in the promppt
Diversy sampling
--- Diversty sampling
Similerit Random tampling
- Random semaling
+
Simlarity semaling
Figure 12: Few-shot results for comparing different sampling strategies on Text-to-SQI. problems of different difficulty levels, with different number of demonstration examples selected for the prompt.


Reviews:
[{"review_id": "BPwRKg6U0X", "reviewer": "Reviewer_ZrTD", "summary": "", "questions": "1. Line 171, by \"binary features\" do you mean one-hot vectors? An example would help illustrate the idea here. \n\n2. Line 197-199, you said \"based on its category, retrieve candidate examples that belong to the relevant partition\". How many candidate examples are retrieved here? Which part of Algorithm 1 does it correspond to? \n\n3. Related, what is the rationale for using difficulty level as a proxy for (syntactic) similarity? I imagine questions in the same category level may be quite different syntactically. Have you investigated other similarity measures? \n\n4. In the zero-shot setting, why does SD not work? \n\n5. Line 496 and in Fig. 3 middle, why does RoBERTa have stronger performance than Ada, which, as a much larger model, should have better performance. Any insights?\n\n6. What are the \"preliminary models\" in Sec. 4.5?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort you have dedicated to reviewing our work, we appreciate your constructive comments, which offer important perspectives for improving the quality of our paper! Below, we address each of your concerns in detail.\n\n> Line 171, by \"binary features\" do you mean one-hot vectors? An example would help illustrate the idea here.\n\nNo, we don't mean one-hot vectors. Our binary feature vectors are more like “bag-of-syntactic-elements”, with each entry being 1 if a syntactic element is present in the SQL query (instead of the count), and 0 if not. For example, “select airportname from airports where airportcode = \"AKO\"” is converted to [1,1,1,1,0,0,...], where the first four entries correspond to the keywords “select”, “from”, “where” and “=”.\n\n> Line 197-199, you said \"based on its category, retrieve candidate examples that belong to the relevant partition\". How many candidate examples are retrieved here? Which part of Algorithm 1 does it correspond to?\n\nWe adopt the difficulty-based categorization of SQL queries based on the implementation of Spider (Yu et al., 2018). We use Spider training split as the pool, which consists of 7,000 instances, from them we found 1,686 easy instances, 2,760 medium instances, 1,446 hard instances, 1,058 extra-hard instances, and 50 erroneous instances which we exclude from the pool. Other datasets use the same training split SQL queries as those in Spider, so the same statistics apply.\nTherefore, given a test instance, we use a preliminary model to generate a draft SQL, and then categorize it based on the aforementioned difficulty levels. For example, If the draft SQL is predicted to be easy, we retrieve the 1,686 easy instances as the candidates, from which we sample diverse demonstrations. This process corresponds to the first “for loop” in Algorithm 1.\n\n> Related, what is the rationale for using difficulty level as a proxy for (syntactic) similarity? I imagine questions in the same category level may be quite different syntactically. Have you investigated other similarity measures?\n\nWe use the difficulty criteria defined and implemented in the Spider (Yu et al., 2018) as the proxy, because their categorization is developed strictly based on syntactic coverage and structure of a SQL query, ensuring that queries satisfying the same conditions are grouped into the same category. For instance, easy SQL queries should contain at most 1 keyword from WHERE, GROUP BY, ORDER BY, LIMIT, JOIN, OR, LIKE, HAVING and no keyword from EXCEPT, UNION, INTERSECT, NESTED and no more than 1 aggregation or column selection. We think this criteria already effectively groups SQL queries with similar syntactic structure and keyword coverage, and we did not explore semantic-based similarity because our main argument is that we can retrieve better Text-to-SQL demonstrations for prompting if we exploit the syntactic information of Text-to-SQL examples. We thank the reviewer for bringing up this question and we will provide more clarifications in the revised version of our paper.\n\n> In the zero-shot setting, why does SD not work?\n\nNot sure which result datapoint the reviewer is referring to. Similarity-Diversity sampling (SD) requires sampling demonstrations, so it only applies to few-shot settings. In zero-shot experiments, we only compare how adding different ontology-related context to the prompt would help.\n\n> Line 496 and in Fig. 3 middle, why does RoBERTa have stronger performance than Ada, which, as a much larger model, should have better performance. Any insights?\n\nWe suspect that the poor performance of ada compared to RoBERTa-base, as shown in Figure 3(a, b), is due to its enhanced capability to retrieve semantically similar examples that are not necessarily syntactically similar, for example, it retrieves examples from the same database cued by similar lexicons, while these examples are likely to have different problem structures therefore would not be helpful demonstrations for solving the current test instance. We will assess the syntactic similarity of all demonstrations retrieved by RoBERTa and ada models to substantiate our hypothesis in the revised version of our paper.\n\n> What are the \"preliminary models\" in Sec. 4.5?\n\nPreliminary models are Text-to-SQL models that can be used to draft the SQL code. We can use any available Text-to-SQL baselines for drafting a SQL query. In Figure 8, we use baselines with different performance (blue lines) to see whether our method can be limited by applying a poor preliminary model. The results show that even with a poor draft (leftmost datapoint), the similarity-diversity approach and the integrated approach are still effective. We will provide more clarification about this in the revised version of our paper. \n\n> The technical contributions and novelty in this paper are limited. It empirically investigates different technical aspects that affect semantic parsing performance in ICL.\n\nWe would like to re-emphasize our contributions. We identified two aspects that were ignored previously when designing prompts for Text-to-SQL tasks using LLMs, which can be extended to other code generation tasks: 1. If the output is code, then the syntactic information of the code is a better representation of the problem structure than the natural language query, therefore serves as a better basis for retrieving demonstrations. Most existing demonstration-retrieval methods, however, rely on query semantics. When retrieving demonstrations based on syntax, we could synergize similarity and the diversity objectives to retrieve examples that are more helpful to solve the test instance; 2. For database related tasks, it's also crucial to incorporate the database ontology (e.g., column type, foreign key constraints, etc.) into the prompt. Last but not least, we propose a procedure to synergize these two augmentations. \n\n> Similar investigations have been conducted in many other tasks already, and there are not novel findings.\n\nIt would be greatly helpful if the reviewer could give a pointer to the similar investigations conducted in other tasks, as we would be glad to include comparisons with them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Semantic parsing is an important NLP task with practical importance, as it allows neural models to interrogate real-world data. Thus, improving in-context learning performance is important for LLMs on this task.", "reasons_to_reject": "* The technical contributions and novelty in this paper are limited. It empirically investigates different technical aspects that affect semantic parsing performance in ICL. Similar investigations have been conducted in many other tasks already, and there are not novel findings. \n\n* The presentation of one of the main ideas in the paper, namely sample selection, is not clear. The diversity and similarity part (Algorithm 1 and associated description in Sec. 2.1) is confusing. \n\n* The findings are not completely convincing or surprising.", "questions_to_authors": "1. Line 171, by \"binary features\" do you mean one-hot vectors? An example would help illustrate the idea here. \n\n2. Line 197-199, you said \"based on its category, retrieve candidate examples that belong to the relevant partition\". How many candidate examples are retrieved here? Which part of Algorithm 1 does it correspond to? \n\n3. Related, what is the rationale for using difficulty level as a proxy for (syntactic) similarity? I imagine questions in the same category level may be quite different syntactically. Have you investigated other similarity measures? \n\n4. In the zero-shot setting, why does SD not work? \n\n5. Line 496 and in Fig. 3 middle, why does RoBERTa have stronger performance than Ada, which, as a much larger model, should have better performance. Any insights?\n\n6. What are the \"preliminary models\" in Sec. 4.5?", "paper_topic_and_main_contributions": "This paper investigates techniques for improving performance of semantic parsing, specifically text-to-SQL, in in-context learning settings. It proposes (1) sampling techniques that take into account both similarity and diversity,  (2) different approaches to include schema information in prompts, and (3) ensemble approaches to aggregate results to tackle the sensitivity issue. \n\nThe evaluation is conducted on 5 different variants of the Spider dataset on Codex and ChatGPT.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ZiHrqyQOU", "reviewer": "Reviewer_FoCC", "summary": "", "questions": "\"Code\" method seems like evaluated only as a baseline setting in zero-shot. What's the point to include this baseline?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper! We have carefully incorporated them in the revised paper. Below, we address each of your concerns in detail.\n\n> The comparison is conducted against a customized baseline, making it challenging to directly align the results with previous work, such as Liu et al., 2023. This limitation also applies when attempting to validate the potential of the proposed demonstration-selection method within a few-shot setting. While I believe that previous in-context demonstration-selection approaches might not be well-suited for this Text-To-SQL task, it is essential to conduct fair comparisons to substantiate this claim with concrete evidence.\n\nWe fully agree that aligning our results with previous work for fair comparisons is crucial for substantiating our claims. It's worth noting, however, that the resource constraints influenced the extent of our experimentation. While we were able to conduct few-shot evaluations for both Codex and ChatGPT, we presented zero-shot results for Codex only due to budget limitations.  Nonetheless, from Codex zero-shot results, we can already see that the baseline (DB as text seq) results on all the Spider variant datasets achieves similar performance compared to Liu et al's ChatGPT results (comparing EX scores) [1], and our proposed ontology-related augmentation can further improve them. Rajkumar et al., 2022 [2] also reports Codex results in zero-shot setting, similar to ours on the Spider dataset. These seem to be the only related prior work that study prompting with Codex/ChatGPT and reported results on Spider or variant datasets at the time of writing. Although Ni et al. [3] and Chen et al. [4] conducted few-shot experiments on Spider, their work diverges from ours by focusing on verification and self-debugging processes, which is an orthogonal investigation and not related to our claims.\nWe plan to include these broader comparisons in our revised paper to provide a comprehensive understanding of our contribution. If the reviewer is aware of additional work that would require a comparison, we're open to including those as well.\n\n[1]: Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability. arXiv preprint arXiv: 2303.13547\n\n[2]: Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv: 2204.00498\n\n[3]: Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin. 2023. LEVER: Learning to Verify Language-to-Code Generation with Execution. arXiv preprint arXiv: 2302.08468\n\n[4]: Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv preprint arXiv: 2304.05128\n\n> The claim that syntactic based retrieval is better than other methods lacks of support.\n> \n> Without refinement, current way of organizing the paper makes me difficult to follow the exact method being proposed.\n\nIt would be greatly helpful if the reviewer could specify which aspect of the claim that they find lacking in support, so that we can address it more effectively in the revision.\nRegarding paper organization, we proposed two prompt augmentations: 1. When selecting demonstrations, code syntax is a better source to extract representations, and introducing diversity in addition to the previous similarity-based retrieval approach helps; (Section 2.1) 2. adding ontology-related context of the database to the prompt helps. (Section 2.2) In addition, we propose a procedure to synergize these two augmentations. (Section 2.3) We will arrange these claims more clearly in the revision.\n\n> \"Code\" method seems like evaluated only as a baseline setting in zero-shot. What's the point to include this baseline?\n\nIf the reviewer is referring to “DB as code-seq” in Figure 1(b), we linearize database as code-sequence for all other prompts (zero or few-shot), here we compare it with “DB as text-seq” in zero-shot setting to show that linearizing database schema to CREATE queries is indeed effective in improving the performance.\n\n> 1. The title can be further refined. Initially, it gives the impression of a technical report that compares various strategies without introducing any advanced approaches. Additionally, the paper proposes enhancements for both zero-shot and few-shot settings, and the mention of few-shot in the title might not be appropriate.\n> \n> 2. I believe there is room for improvement in conveying the message of the paper. While the current approach heavily relies on descriptions about the proposed method, incorporating simple examples would greatly enhance clarity. For instance, a example of the structured predictionas in Section 2.1 would be helpful for better understanding. Additionally, after the clustering process, including examples of the selected demonstrations will provide readers with a clearer insight into the proposed method.\n> \n> 3. There are certain details in experiments can be added, like how many testing samples in total, temperature setting for GPT, etc.\n> \n> 4. I have to switch back several times to align the terminology used in experiments like Schema Augmentation, Structure Augmentation and Semantic Augmentation with the one in methodology part. It will be good if highlight those terms can be highlithed.\n\nWe appreciate the reviewer for bringing up these feedbacks! We will revise the paper to address these issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed demonstration-selection approach, which takes into account syntactic structure, diversity, and similarity, shows promise and can be of inspiring. It has the potential to provide valuable insights and inspiration for further advancements in the area of LLM based Text-To-SQL tasks.\n2. The proposed method demonstrates notable improvements on semantic parsing datasets.", "reasons_to_reject": "1. The comparison is conducted against a customized baseline, making it challenging to directly align the results with previous work, such as Liu et al., 2023. This limitation also applies when attempting to validate the potential of the proposed demonstration-selection method within a few-shot setting. While I believe that previous in-context demonstration-selection approaches might not be well-suited for this Text-To-SQL task, it is essential to conduct fair comparisons to substantiate this claim with concrete evidence.\n2. The claim that syntactic based retrieval is better than other methods lacks of support.\n3. Without refinement, current way of organizing the paper makes me difficult to follow the exact method being proposed.\n", "questions_to_authors": "\"Code\" method seems like evaluated only as a baseline setting in zero-shot. What's the point to include this baseline?", "paper_topic_and_main_contributions": "This submission presents a novel approach to enhance the Text-To-SQL task by incorporating two key components: demonstration-selection and schema augmentation. It conducts experiments on semantic parsing datasets to prove the effectiveness of proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The title can be further refined. Initially, it gives the impression of a technical report that compares various strategies without introducing any advanced approaches. Additionally, the paper proposes enhancements for both zero-shot and few-shot settings, and the mention of few-shot in the title might not be appropriate.\n\n2. I believe there is room for improvement in conveying the message of the paper. While the current approach heavily relies on descriptions about the proposed method, incorporating simple examples would greatly enhance clarity. For instance, a example of the structured predictionas in Section 2.1 would be helpful for better understanding. Additionally, after the clustering process, including examples of the selected demonstrations will provide readers with a clearer insight into the proposed method. \n\n3. There are certain details in experiments can be added, like how many testing samples in total, temperature setting for GPT, etc.\n\n4. I have to switch back several times to align the terminology used in experiments like Schema Augmentation, Structure Augmentation and Semantic Augmentation with the one in methodology part. It will be good if highlight those terms can be highlithed.\n"}}, {"review_id": "xrpiwh1itz", "reviewer": "Reviewer_rZ1M", "summary": "", "questions": "1. How does the Codex perform when retrieving demonstrations with the oracle SQL query of the test instance?\n2. How many experiments are run to achieve the majority voting for one test instance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and valuable suggestions! They have been incredibly useful, and we will integrate them into the updated manuscript. In the following, we would like to address your comments and questions.\n\n> The authors should be more careful when saying \"we propose\". (it does not sound like a problem to me once some editing is done). (1) In Line 224, the authors say \"We propose ... code sequence ... CREATE query\". The CREATE query for database schema was proposed by [1]. The authors should be aware of this work as it is cited in the introduction section. (2) Retrieve diverse demonstrations for semantic parsing has been proposed in this work [2], although it is not focusing on text-to-SQL.\n\nWe would like to thank the reviewer for pointing this out and we agree that we should change the phrasing “we propose” and mention Rajkumar et al., 2022 regarding CREATE query approach. We also appreciate the reviewer for the second reference which we overlooked, and we will add it to the revised manuscript.\n\n> How does the Codex perform when retrieving demonstrations with the oracle SQL query of the test instance?\n\nWe have not conducted experiments of Codex retrieving demonstrations with the test-set oracle SQL queries directly, and will conduct them if we regain access to Codex. But with draft SQL, we are able to obtain 74% accuracy on difficulty-level prediction, indicating that for 74% of the test instances, we retrieved the same demonstrations as those retrieved using oracle SQL and therefore leading to the same results.\n\n> How many experiments are run to achieve the majority voting for one test instance?\n\nWe obtain 7 experiment results (4-10 shots) to achieve the majority voting for one test instance. We will add this clarification in the revised manuscript.\n\n> The best fine-tuned system was [1], which was released more than 3 months ahead of the EMNLP2023 deadline. [1] Li, Haoyang, et al. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n\nWe appreciate the reviewer for pointing out the missing reference, and we will add it to the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper compares various demonstration retrieval strategies for text-to-SQL with in-context learning and reveals that the most preferred demonstrations are to consider both similarity and diversity.\n2. The paper also studies how to represent schema-related knowledge in the zero-shot prompt designs. \n3. The experiments on multiple text-to-SQL datasets demonstrate the generalization of the findings.\n4. The paper is well-written and easy to follow.", "reasons_to_reject": "1. The authors should be more careful when saying \"we propose\". (it does not sound like a problem to me once some editing is done).\n(1) In Line 224, the authors say \"We propose ... code sequence ... CREATE query\". The CREATE query for database schema was proposed by [1]. The authors should be aware of this work as it is cited in the introduction section.\n(2) Retrieve diverse demonstrations for semantic parsing has been proposed in this work [2], although it is not focusing on text-to-SQL.  \n\n[1] Rajkumar, Nitarshan, Raymond Li, and Dzmitry Bahdanau. \"Evaluating the text-to-sql capabilities of large language models.\" arXiv preprint arXiv:2204.00498 (2022).\n[2] Levy, Itay, Ben Bogin, and Jonathan Berant. \"Diverse demonstrations improve in-context compositional generalization.\" arXiv preprint arXiv:2212.06800 (2022).", "questions_to_authors": "1. How does the Codex perform when retrieving demonstrations with the oracle SQL query of the test instance?\n2. How many experiments are run to achieve the majority voting for one test instance?", "paper_topic_and_main_contributions": "The paper explores prompt design strategies for text-to-SQL with in-context learning. The authors propose to retrieve demonstrations with SQL syntactic structure. Given a test instance, the proposed method first generates a preliminary SQL query and uses the preliminary SQL query to retrieve demonstration examples. For the retrieval strategy, the authors propose to consider both the diversity of demonstrations and the similarity between demonstrations and the test instance. The paper also studies how to represent schema-related knowledge in prompt designs in a zero-shot scenario. \nBy combining schema-related knowledge and the demonstration retrieval method, the proposed method outperforms the previous state-of-the-art in-context learning methods and fine-tuned methods. The findings in the paper will benefit future study text-to-SQL.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Missing_References": "The best fine-tuned system was [1], which was released more than 3 months ahead of the EMNLP2023 deadline.\n[1] Li, Haoyang, et al. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023."}}]

Author rebuttal:

Meta-review: This paper proposes a novel prompt design strategy for text-to-SQL with in-context learning. The proposed approach first retrieves demonstrations for a test instance by generating preliminary SQL query and both diversity and similarity between demonstrations and the test instance are considered. This work introduces approaches to include schema-related knowledge in prompts, and ensemble approaches to aggregate results to tackle the sensitivity issue.

Combination of all aforementioned information, outperforms the SOTA of in-context learning methods and fine-tuned methods. The findings of this work could benefit the community for studying text-to-SQL.
Decision: Accept-Findings