Benchmarking and Improving Text-to-SQL Generation under Ambiguity
Authors: Adithya Bhaskar
Tushar Tomar
Ashutosh Sathe
Sunita Sarawagi
Venue: EMNLP 2023 Main (2023)
Keywords: Semantic Parsing
Text-to-SQL
Ambiguity
Beam Search
ChatGPT
LLMs for Text-to-SQL
TL;DR: We introduce a benchmark on Ambiguous Text-to-SQL conversion and significantly improve upon SOTA approaches with a novel decoding method.

Abstract: Research in Text-to-SQL conversion has been largely benchmarked against datasets where each text query corresponds to one correct SQL.  However, natural language queries over real-life databases frequently involve significant ambiguity about the intended SQL due to overlapping schema names and multiple confusing relationship paths. To bridge this gap, we develop a novel benchmark called AmbiQT with over 3000 examples where each text is interpretable as two plausible SQLs due to lexical and/or structural ambiguity. 

When faced with ambiguity, an ideal top-$k$ decoder should generate all valid interpretations for possible disambiguation by the user. We evaluate several Text-to-SQL systems and decoding algorithms, including those employing state-of-the-art LLMs, and find them to be far from this ideal. The primary reason is that the prevalent beam search algorithm and its variants, treat SQL queries as a string and produce unhelpful token-level diversity in the top-$k$.

We propose LogicalBeam, a new decoding algorithm that navigates the SQL logic space using a blend of plan-based template generation and constrained infilling. Counterfactually generated plans diversify templates while in-filling with a beam-search that branches solely on schema names provides value diversity. LogicalBeam is up to $2.5$ times more effective than state-of-the-art models at generating all candidate SQLs in the top-$k$ ranked outputs. It also enhances the top-$5$ Exact and Execution Match Accuracies on SPIDER and Kaggle DBQA.

Benchmarking and Improving Text-to-SQL Generation under Ambiguity
Adithya Bhaskar'
Tushar Tomar
Ashutosh Sathe
Sunita Sarawagi
IIT Bombay
& Princeton University
Abstract
Mooney, 1996; Tang and Mooney, 2000: Scholak
et al., 2021a; Wang et al., 2020; Rubin and Be- rant, 2021; ; Xie et al., 2022; Arcadinho et al., 2022; Zeng et al., 2022; Scholak et al., 2021b; Pourreza and Rafiei, 2023). Popular benchmarks driving such research, including WikiSQL. (Zhong et al., 2018), SPIDER (Yu et al., 2018), its robust per- turbations (Chang et al., 2023), and even "in-the- wild" benchmarks such KaggleDBQA (Lee al., 2021) and SEDE (Hazoom et al., 2021) all asso- ciate one correct SQL with text. Meanwhile, am- biguity is prevalent in real-life databases - par- ticularly the ones obtained by integrating several data sources for data analysis, where natural lan- guage interface is most in demand. The sources of ambiguity several inherent ambiguity of nat- ural language, the user's ignorance of table/column names, overlapping strings in column names, under- specified clauses, and confusion about whether ag- gregates are pre-computed, or a join required. Hazoom et al. (2021) observe that up to 87% of queries on the stack exchange database are under- specified, and Wang et al. (2022) mention that 1 11% of queries exhibited ambiguity in column names. Although prior work has brought up ambiguity, there is no publicly available benchmark with am- biguous queries, nor a comprehensive evaluation of systems under ambiguity.
Research in Text-to-QLL conversion has been largely benchmarked against datasets where each text query corresponds to correct SQL. However, natural language queries over real- life databases frequently involve significant am- biguity about the intended SQL due to lapping schema names and multiple confusing relationship paths. To bridge this gap, we de- velop novel benchmark called AmbiQT with over 3000 examples where each text is inter- pretable as two plausible SQLs due to lexical and/or structural ambiguity.
When faced with ambiguity, an ideal top-k de- coder should generate all valid interpretations for possible disambiguation by the user (E))- gohary et al., 2021; Zhong et al., 2022). We evaluate several Text-to-SQL systems and de- coding algorithms, including those employing state-of-the-art LLMs, and find them be far from this ideal. The primary reason is that the prevalent beam search algorithm and its vari- ants, treat SQL. queries as string and produce unhelpful token-level diversity in the top-k.
We propose LogicalBeam. new decoding al- gorithm that navigates the SQL. logic space us- ing blend of plan-based template generation and constrained infilling. Counterfactually gen- erated plans diversify templates while in- in-filling with beam-search, that branches solely schema names, provides value diversity. Log- icalBeam is up to 2.5x more effective than state-of-the -fr models at generating all candi- date SQLs in the top-k ranked outputs. also enhances the top-5 Exact and Execution Match Accuracies on SPIDER and Kaggle DBQA'.
Our first contribution is to bridge this gulf by developing a benchmark, AmbiQT, that tests per- formance under ambiguity in the context of current models. AmbiQT includes over 3000 examples, each associating a natural question on a database with two valid SQLs. Inspired by our experi- ence with several real-world datasets, we target four types of ambiguity spanning both lexical (am- biguous column and table names) and structural (whether a join necessary, and an aggregate is pre-computed) ambiguity. The benchmark gen- crated via a combination of ChatGPT (OpenAI, 2022) based synonym generation and perturbation,
1 Introduction
Research Text-to-SQL generation has focused on scenarios where each natural language ques- tion is associated with one correct SQL (Zelle and
We release AmhiQT and LogicalBeam's implementation publicly at https:/// htw.b//gtttbb.../// thub. cwm/ot//eeeeeeeeeeeeec testzer0/ A Equal Contribution, Work done while AB was at IIT Bombay. Correspondence to: <adi chabschaacee@gmai........ com-

---

and standard rule-based perturbation.
Table concert
When faced with ambiguity, an ideal Text-to- SQL system should incorporate all valid alterna- tives in their top-k SQL outputs, for user resolu- tion. We show that present approaches, ranging from T5-3B (Raffel et al., 2019) to SOTA mod- els, fail to generate all ambiguous outputs with any decoding strategy, including beam search and prromitty sampling methods such Nu- cleus (Holtzman al., 2020) and Typical sam- pling (Meister al., 2023). Most outputs are small lexical tweaks of the top choice, bringing about little meaningful diversity in SQL structures or schema alternatives. Even SOTA LLMs like Chat- GPT (OpenAI, 2022) suffer from this issue.
Show the stadium of concents stadium has hosied
SingeriD Stodiumlo Location Name:
StadiumDD Capacity
Location
Name
7500
XY2
ABC
Table stadium
Texi-oo-SLL parser
SELECT 12. name, COUNT (*)
FROM concert AS 21 JoIn stadiun as '2
1l. StadiunID StadiunDD
GROOP BY 1i StadiunnD
Figure I: A Text-to-SQL system converts user ques- tion to an SQL query, conditioned on the database schema and/or content.
Benchmarks. Popular benchmarks for the Text- to-SQL task are WikiSQL (Zhong et al., 2018) and SPIDER (Yu et al., 2018). A few others have been proposed recently to capture real-world sce- narios, such as KaggleDBQA (Lee et al., 2021), SEDE (Hazoom et al., 2021), and EHRSQL. (Lee et al., 2022). They all attach though some of them mention the problem of am- biguity in real-world datasets. Dr. SPIDER (Chang et al., 2023), designed to test the robustness of ex- isting models, perturbs either the text or schema of SPIDER but still assigns one SQL per text.
To remedy the lack of diversity, propose new decoding algorithm, LogicalBeam, that allo- cates branching to explore underlying logical vari- ants of the SQL rather than the string form. We catalog the errors of T5-3B (Raffel et al., 2019) on the SPIDER dev split and use our insights to en- courage targeted types of diversity - the number of JOINs and selections, and table/column names.
SQL per text,
Our main contributions are:
We develop AmbiQT, the first benchmark that tests performance under four types of ambiguity over 3000+ examples.
Ambiguity in SQL Although ambiguity has been studied in other fields of NLP (Pilault et al., 2023; Li et al.. 2022; Futeral et al., 2022), it has been unexplored in the context of semantic parsing. Ambiguity in SQL arising from related column names is discussed (Wang et al., 2022), but they only consider column ambiguity. Their method of recognizing ambiguous queries depends on la- beling words of the text and does not generalize to other kinds of ambiguity. To the best of our discern- ment, AmbiQT represents the first open benchmark for testing coverage of ambiguous alternatives.
We show that SOTA methods, including fine- tuned T5-3B, RESDSQL (Li al., 2023), Ope- nAI Codex, and ChatGPT, provide poor repre- sentation ambiguity despite their high accu- racy on conventional benchmarks.
We present LogicalBeam, a two-step algorithm that generates plan-based templates with counter- factually controlled plan diversity and fills them via beam search that branches only on schema names.
We show that LogicalBeam consistently in- creases the fraction of time when all gold SQLs get generated in the Top-5 choices by 1.5 2.5x over the baselines across the board AmbiQT.
Diverse Decoding. Prior work has critiqued the lack of meaningful diversity beam-seardh out- puts (Fiikel et al., 2006: Gimpel et al., 2013; et al., 2016; Li and Jurafsky, 2016). In response, many fixes have been proposed. Some proposals attempt to restrict the tokens sampled, using strate- gies like Nucleus sampling (Holtzman et al., 2020), Truncated Sampling (Hewitt et al., 2022), and Typ- ical Sampling (Meister et al., 2023), while some rely on Template-Bas Based decoding (Wiseman et al., 201 8; Zhang et al., 2022; Fu et al., 2023; Elgohary et al., 2020; Awasthi et al., 2022). A third approach is to generate prefix with high diversity first, then
2 Background and Related Work
A Text-to-SQL. model takes as input question ex- pressed as natural language text and a database schema comprising of table and column names, and outputs an SQL program y which can be ex- ecuted against the database to answer the user's question. Figure shows an example. The training data for the task comprises (text, schema, SQL) triplets spanning multiple distinct databases.

---

Kind of
Example
Count
ambiguity
Question Text
SQL #1
SQL #2
Column
List the ids of all students.
SELECT roll_number FROM students
SELECT udmission__umber FROM students
1240
Ambiguity (C)
Table Ambiguity (T)
How many singers do have?
SELECT COUNT(*)
SELECT COUNT(*) FROM performer
1417
FROM artist
SELECT t2. maker,, t1.model FROM
Join Ambiguity (J)
What are the makers and models?
SELECT maker, model
288
model JOIN model_maker
FROM model
t1.model_id t2. model_id
Precomputed
Find the average weight SELECT AVG(wiigh,) pettype for each pet type.
SELECT avg_weight, pettype FROM pets_weight.
101
Aggregates (P)
FROM pets GROUP BY pettype
Table I: The AmbiQT benchmark. For each question, we list two valid SQL queries as per the schema. The schemn is not shown here, but the ambiguity in it can be inferred based on the two SQL queries.
generate the rest of the sentence with lower diver- sity. Narayan et al. (2022) follow this recipe but focus incorporating diverse entity orders in text summarization.
name of s in a one-shot manner. Appendix A fur- nishes more details of the prompt. We then modify by replacing with two columns c1, C2, and we use y to generate two queries y1, V2 where all men- tions of e are replaced with e1 in Y1 and with c2 in Y2. An example appears in the first row of Ta- ble 1. We do not reuse e because the SPIDER dataset often contains column names verbatim in the question, and that would violate our attempt at keeping the two options at similar relevance levels. We modify one column at a time and generate up to 3 examples from each original entry.
3 AmbiQT: A Benchmark of Ambiguous Text-to-SQL Conversion
AmbiQT is constructed so that each text query has two distinct valid SQL interpretations. Motivated by our experience working with real-life databases, we designed AmbiQT to encompass four types of ambiguity. Each entry is designed so that both alternatives have similar relevance the question, and well-calibrated decoding method is expected rank them close by in their outputs.
Table Ambiguity (T). Table name ambiguity is common in databases obtained by integrating mul- tiple data sources, as in web tables (Cafarella al., 2008; Pimplikar and Sarawagi. 2012). Here again, we prompt ChatGPT to generate two alternate names for each table. We then modify one SQL to generate two candidates Y1. as shown in Table 1.
We create AmbiQT by modifying the SPIDER (Yu et al., 2018 88 dataset, and use ChatGPT (Ope- nAI, 2022) to aid with the creation. In each case, we modify the schema instead of the text as that pro- vides greater control over the modification process. We explain the kinds of ambiguity in AmbiQT be- low and portray examples of each in Table
Join Ambiguity (J). In production databases, a logical table is often vertically partitioned across several tables for efficient clustered access (Stone- braker al., 2019). Column names overlapping across tables leads to Join Ambiguity. Suppose we have two tables: (1) person with columns id, name, email_address, and (2) person_details with columns id, postal_address, photo. A question asking for a person's name and ad- dress is ambiguous on whether a JOIN with the person_detaile is necessary. We expose such ambiguity by modifying the schema follows.
Column Ambiguity (C). Unlike the SPIDER benchmark where column names usually appear verbatim in the question text (like born state for the column born_state), when users unaware of the schema pose natural question, they introduce column ambiguity (Wang al., 2022). For exam- ple, "What is the capacity of 02 Arena?" could be ambiguous the schema has separate columns for standing and seating capacity. Likewise, a query on the number of under-nourished children is ambigu- ous if we have different columns for weight children" and "stunted growth in children".
Consider (x, s, triplet. Suppose y involves selecting two more columns C1, C2 ... not nec- essarily in the same order, from a table Suppose further that ej is not a primary key of t. We create a table called t that includes just the primary
To simulate column ambiguity, for each text x, schema s, and SQL y in SPIDER, we prompt Chat- GPT to generate two synonyms for each column

---

Question:
high school students and corresponding of friends.
Gold Queries
FROM friend JOIN highschooler on tl.id c2.id GROUP tt.id 2. SELECT 12. 2.given_ gagieennnmee, count(") FROM friend AS JOIN highschooler t1.id GROUP tl.id
1. SELECT 12.full name. count(*) FROM
Outputs of T5-3B with Beam Search
1. SELECT 11. gllggven___ng (o)nt(") FROM highschooler AS LI JOIN friend AS 12 wm..m.l..dd... = GROUP BY 11.id
2. SELECT tl.given namee count(")) FROM highschooler AS JOIN friend AS (2 on tl.dd 12.id GROUP BY highhseeelleer
3. SELECT fl.given_name name, count(") FROM highschooler AS tl JOIN friend AS 12 on t1.id= (2.rrcen id GROUP BY gl.gddd
4. SELECT tl name, count(") FROM bighschooler AS JOIN friend AS 12 on t1.id 12.id GROUP BY tl.id
5. SELECT tl.giving name, count(") ") FROM highschooler AS tl JOIN friend AS 12 tl.id = 12.id GROUP BY d1.d
Figure 2: Beam Search works well when targeting only one output, but leads to superficial diversity, for example via different grouping and erroneous variants of column names.
key ph, of f, and Cj. The first alternative yı is y and the second alternative y2 uses a join over and f_c,, with everything else staying the same as y.
and ChatGPT. A few anecdotes for each kind of ambiguity are shown in Appendix F. The reason for the failure is that Beam Search tends to produce outputs that are minor tweaks of the best hypoth- esis, as also corroborated by prior work (Finkel et al., 2006; Gimpel al., 2013; Li et al., 2016; Li and Jurafsky, 2016). One example from the 'C' split of AmbiQT that illustrates this is displayed in Figure 2. Recent iiversity-promoting decod- ing strategies like Nucleus (Holtzman et al., 2020) and Typical (Meister et al., 2023) sampling are de- signed for natural language and are ineffective for capturing the structural diversity that SQL variants require. These observations motivated the design of our inference algorithm, LogicalBeam.
Precomputed Aggregates (P): This ambiguity is particularly common in data warchouses such as Data Commons which pre-aggregate certain vari- ables. For instance, the "total rice production" of state might refer to the column rice_production of state rather than sum over it. Text-to- SQL models have a bias toward introducing a sum().. by clause every time total ap- pears in the text. The non-aggregated alternative is usually missing in the top-k options. We incorpo- rate this ambiguity as follows.
For each (x, s, y), where y has least one ag- gregate, we construct a new table t'. For each aggregate A over column e in y, we add to the columns A' for all A' E {avg, sum, min, max]; J. and the columns grouped by in y. For count (*) we add column called number. We get two gold queries, the original y and second with the group- by replaced by a direct SELECT on E' as shown in the example in Table 1. We also support aggregates across multiple tables but skip the details here.
5 Our method: LogicalBeam
LogicalBeam attempts to induce meaningful di- versity, while steering clear of vacuous forms of diversity in the formatting of the SQL. We first at- tempt to understand the type of logical diversity required by analyzing the errors of the top-1 output of T5-3B the SPIDER benchmark.
4
Are Existing Text-to-SQL systems resilient to ambiguity?
The mistakes of the top-1 output are cataloged in Table 2. Apart from the column selection order, which is arguably not a serious error, the top four errors are a wrong number of joins, columns, and incorrect column and table names. A large fraction of the errors involves the "skeletal structure" of the SQL, whereas vanilla Beam Search exhibits little diversity in the SQL structure. Most of its diversity around generating alternate forms of string liter- als, tweaking comparison orders, or swapping the names of temporary variables (like t1 with t2).
We evaluate several SOTA Text-to-SQL. models and decoding algorithms on their ability to generate the alternatives of AmbiQT in their top-k outputs. Descriptions of the systems compared and evalu- ation metrics appear in Subsection 6.2. Table 3 features the results we obtained.
For all systems, the top-5 outputs contain both outputs only for a small percentage the instances. To investigate the reasons for their poor coverage, we manually inspected several outputs of T5-3B

---

First, we preface template with plan declar- ing the structural properties of the SQL where di- versity is desired. Based our error analysis in Second, counterfactually perturb the counts in the plan follows. We generate the top- choice template without any constraints (say, with joins and s selections). We then gener- ate four diverse plans by searching in the neigh- borhood of the most likely predicted structure as (j (j-1, 1), (j 1,,), (i - 1), (j. 1). We skip invalid combinations < 0, 3, or S 0). We also explored sampling j, based on predicted probabilities, but these were extremely skewed.
Contrib-
Fror Type
ution (%)
Correct Output
70.31
Wrong selection order
7.44
Missing JOIN(s)
6.09
Missing column(s)
3.48
Extra JOIN(s) introduced
2.80
Incorrect column or table names
2.51
WHERE clause missing incorrect
1.35
Extra column introduced
1.06
ORDER BY clause missing or incorrect
0,48
GROUP BY clause missing or incorrect
0.39
Wrong comparison
0,29
DISTINCT missed or introduced Other
0.10
Finally, for each plan (enforced as a prefix), we use greedy decoding to generate the template. The decoding algorithm was good at generating tem- plates per the specified plan.
3.68
Table 2: A catalog of errrrs on the SPIDER dev split, based on Exact Match (EM), corresponding to the top-1 output from Beam Search with beam width of 25. Most errors stem from an incorrect number of JOINs SELECTions, with incorrect schema names being concern as well.
Thus, at the end of the template generation phase. we have at most five templates.
5.2 Template filling with Diverse Schema
Here we fill diverse column names and table names in the generated templates. We use beam search to this end but enforce adherence to the template. We track our position in the template during infilling, If the next token is expected to not be part of a table or column name, we disallow the model from exploring anything apart from the highest-scoring next token. Otherwise, we allow it branch in the next decoding step. However, we restrict its options to a whitelist of tokens computed beforehand by enumerating the columns/tables from the schema. The pseudocode of our Restricted Infilling method is presented in Algorithm 1.
These observations drove us to design a two- stage approach. In the first stage, we generate di- verse SQL skeletons (templates) to capture struc- tural diversity, and in the second fill in the tem- plate with echema-diversss alternatives. We illus- trate our approach in Figure 3.
5.1 Plan-based Template Generation
A template of an SQL query abstracts away the names of the tables and columns of the SQL query, string literals, and constants, so that only the struc- tural components (SELECTs, GROUP BYs, JOINs, comparisons and so on) remain. On the train split of SPIDER, we convert the gold SQL to tem- plate by a simple rule-based replacement of schema names (details in Appendix E) and use it to train a Texplao-Temppate model. However, the top-k tem- plates found via beam search this model again lacked logical diversity. One example is shown by Figure in Appendix D. We thus explored a more deliberate mechanism induce diversity following these three steps:
The next challenge is how rank the SQLs from the diverse templates and select the top-5. We ini- tially attempted rank based on the product of probabilities of the template and in-filling steps. However, the probability distribution of the mod- els we worked with was extremely skewed for example, top-p sampling with p 0.9 produced the same template in all infillings over 70% of the time. Combined with the well-known lack of cal- ibration of neural sequence models, we found it better to simply choose the top-2 2 SQLs from each template, along with the top from vanilla beam- scarch without any templates. After filtering out duplicates, the top-5 queries in the list are returne..
Table we chose to induce diversity on the num- ber of JOINs and final SELECTions. Thus, for a given input question, we output a plan followed by a template as:
Experiments
We present extensive comparisons of several State- of-the-Art Text-to-SQL. models and decoding methods on AmbiQT in the following sections. We then show that LogicalBeam can be helpful even
<J> joins
<S> selects <TEMPLATE>
The left yellow box in Figure 3 shows one such plan prefixed template.

---

Algorithm 1: Pseudocode for one Beam Extension step of the Restricted Fill-In Algorithm
Data: Ream width A; current hypotheses and scores (y1, 81), (Y2, 82), (t), template t, set column names and table names T
Result: The next hypotheses with scores (1, (8),))
<-
+ getPossibleefirsttoks())
getfossiblefirsttioss(7)
for (: = 1; l; ==++))
then
/* If this hypothesis violates the template, don't extend it.
*/
continue;
ncls <- getNextTokClass(yy.
/* Check if we expect to start column/table name next.
*/
nels E (column, table) then
18 Allow branching, but restrict options to whitelist.
k/
H. < BeExxenssoonkiittssoreess...
else
/* Disallow branching by only choosing the top-scoring extension
*/
H. + hetTTppEttensinniithccorreyyy
end
H
H IH;
return geTTgkhihothhsssssttttsssr/////
in the absence of ambiguity. We also present detailed ablation study of LogicalBeam, and dis- cussion of its use-cases and shortcomings.
one-shot manner using an example outside of AmbiQT. One-shot prompting was required to get ChatGPT to adhere to the output format. More details can be found in Appendix A. We also show in Appendix В that alternate prompts tried by prior works (such as (Liu et al., 2023)) are inefficient getting ChatGPT to cover all possibilities.
6.1 Implementation Details of LogicalBeam Both stages of LogicalBeam are fine-tuned ver- sions of Flan T5-3B (max length 512), with an Adafactor (Shazeer and Stern, 2018) optimizer (learning rate le 4, and no decay). The models were trained for roughly 300 epochs each, with checkpoint selection based on the highest Template Match and Exact match, respectively (on the vali- dation set, with greedy decoding). Our datasets for the models consist of one-to-one maps of each ex- ample from SPIDER, with, e.g., the SQL query replaced by the corresponding template for the Eext-to-Template model. We use the Hugging- Face Logi t for the Template-Infin Infillinn model, which allows us to modify logits at each de- coding step. We set all the disallowed tokens' logits to 00 implement the restricted beam search.
OpenAl Codex (Codex):. We use few-shot prompting with the code-davinci-000 version of OpenAI Codex (Chen et al., 2021). This is the most capable Codex version at the time of writing. More details are provided in Appendix A.
RESDSQL (RSQL): Among approaches that do not use ChatGPT/GPT-4, RESDSQL (Li et al., 2023) the best-performing method on SPIDER at the time of writing. We use its 3B variant (the most potent one) for comparison but turn off the NatSQL (Gan et al., 2021) representation, as it is orthogonal to our approach and can be used with it. T5-3B (T5-3B):. We use the T5-3B checkpoint from the PICARD (Scholak et al., 2021b) reposi- tory that fine-tunes T5-3B on SPIDER. By default, Beam Search for T5-3B.
6.2 Methods Compared
We compare with the following models. All use Beam Search with beam width of 10 unless oth- erwise specified. For T5-3B (one of the best- performing baselines), alternate decoding algo- rithms are also included in the comparison.
T5-3B with Top- sampling (T5-3B-k): At each sample from the top-50 to-
step of decoding, kens, i.c. using top-k sampling with h 50. T5-3B with Nucleus/Top- Top-p Sampling (T5-3B-p): At each step of decoding, we sample from the top-p tokens that account for 90% of the probability mass as proposed in (Holtzman et al., 2020).
ChatGPT (CGPT): We prompt ChatGPT for its top five choices given the question and schema in
'https: / http://hgggggaaae/eeeee acce http://wca..cccccceetee//aate//t co/docs/transfomeert
TS-3B with Typical Sampling (T5-3B-T): Typ-

---

average age each that are over 40
Top
Prefix forcing defines template attributos
Template Generation defines the structure
joins selects SELECT
column, AVG(column) FROM
table WHERE column number
Counterfactual template
1 joins selects SELECT co colum AVG(columm) FROM table as t1 JOIN table as t2 ON co column cal umn WHERE
generation (+1 join)
GROUP
SELECT born_sstate, (age) FROM head WHERE age 40
Template Infilling determines the
Non existing column/lable names are pruned out, adherence to the template enforced, and branching is disallowed at non-coluumvlable posilions.
SELECT
SELECT t1.born_gge 1i.bor_statee state
SELECT (1.borm_staae saalsslaeee.. ...
t1. 21.born_state, AVG(t2.age) FROM head AS t1 JOIN
SELECT
SELECT birthplace
SELECT t1 bom_state
employee AS t2 e1.emppid t2.emp_id
SELECT birth strth_
SELECT birthplace, AVG(age)
GROUP
ti.born_state
Figure 3: Our approach in its entirety. counterfactual template generation step provides template diversity via Prefix Enforcement. Constrained infilling generates content diversity by restricting branching and enforcing template adherence.
ical Sampling (Meister et al., 2023) is another re- cent diverse decoding algorithm for enforcing nat- ural diversity. This algorithm uses parameter, typical_p, similar to the top_p of Nucleus Sam- pling. Following (Meister et al., 2023), set typical_p to 0.9.
Beam Width
Number of Outputs
30
34.4
30
29.5
28 4
21.9
20
20
21
Flan T5-XL (F-T5-3B): This is the FLAN vari- ant the T5-3B model, fine-tuned with the same PICARD code as the T5-3B model above.
15.9
15.9
16.8
12.9
11.9
12.9
16.4
14.3
10
11.7
LogicalBeam. For both stages we fine-tuned sepa- rate Flan T5-3B (Chung et al., 2022) models. We use a learning rate of 10 4 and an effective batch size of 810 via gradient accumulation in both cases. Evaluation Metries. We present two types of accu- racies (i) EitherinTopK that checks if either of the gold queries feature in the top-5 outputs (ii) Both- InTopk- that checks both gold queries feature in the top-5. We only report the Execution Match (EXM) accuracies for each. The numbers of Exact Set Match are given in Appendix C.
10
15
20
5
10
15
Figure 4: The coverage only increases slightly with more outputs, and decreases with increasing beam width. The x-axis varies the controlled hyperparam- eter, while the y-axis reports coverage.
on the SPIDER dataset, RESDSQL sees its cover- age plummet under ambiguity. We observed that it often produced outputs that corresponded to neither of the alternatives. This behavior was also exhibited by T5-3B, by using aggregates such as max (avg_age). Though outputs produced this way syntactically correct, they do not corre- spond any meaningful question.
6.3 Overall comparison on AmbiQT
We present the results of the system comparison in Table 3. We make the following observations:
State-of-the-art Text-to-SQL models cannot handle ambiguity: Existing approaches. includ- ing T5-3B, CharGPT, and RESDSQL among oth- ers, fail to cover both alternatives Top-5 even when they perform reasonably under the EitherIn- TopK heading. Surprisingly. despite being SOTA
Beam-search gives unhelpful token-level di- versity Although may seem like increasing the beam width allows greater exploration and thus greater diversity, this is not the case. As Figure 4

---

Kind of Ambiguity
CGPT
Codex
RSQI.
F-T5-3B
T5-3B
T5-3B-k
T5-3B-p
T5-3B-T
LogicalBeam
EitherInTopK (%)
C
52.7
55.6
51.0
60.7
59.0
52.0
51.7
48.1
66.6
T
55.7
59.8
33.8
59.7
57.9
49.1
48.7
43.6
67.3
J
77.8
83.7
68.8
86.1
86.8
80.6
79.9
76.7
87.2
P
57.4
77.2
42.6
49.5
58.4
55.4
53.5
51.5
64.4
Bothin Topk (Coverage) (%)
22.7
10.4
10.8
8.7
11.7
2.5
0.0
28.0
37.3
14.3
15.1
21.9
7.1
1.1
42.6
15.6
43.8
0.0
22.6
27.8
2.8
1.7
0.0
59.4
P
8.9
24.7
8.9
2.9
15.8
4.0
3.0
0.0
24.8
Table 3: The results of all compared systems AmbiQT portrayed by Execution Match (EXM) accuracy in the Top-5 outputs. LogicalBeam usually performs the best under the BitherInTopk heading, except for Precomputed Aggregates. More importantly, LogicalBeam consistently outperforms all other systems under the BothInTopk heading. This shows the capacity of LegicalBeam to capture greater meaningful diversity in its outputs.
Kind Ambig- uity
+Schema Diversity LogicalBeam)
mantic parsing under ambiguity since the model produces skewed token probabilities. This causes the sampled hypotheses often identical as seen the anecdotes in Figure 5 in the supple- mentary material.
Single stage
Two stages
+Template Diversity
Toph (%)
C
64.0
65.9
65.9
66.6
T
60.7
66.2
65.0
67.3
LogicalBeam yields substantial accuracy in covering both ambiguous outputs. Its BothIn- TopK accuracy is almost 2.5x better in the case of Column Ambiguity than T5-3B, and 2x in the case of Table ambiguity, It outperforms other systems by a huge margin. The accuracy under Join Ambiguity increases by 1.4x over Codex (the next-best method) and is over 2x better than any other method. For Precomputed Aggregates, LogicalBeam is once again the best system, and surpasses everything apart from Codex by over 1.5x. We also convincingly beat ChatGPT and OpenAl Codex across the board coverage.
J
86.8
88.5
87.1
87.2
P
58.4
62.4
63.4
64.4
Bopk (Coverage) (%)
16.0
28.0
T
25.3
28.4
28.2
42.6
54.5
54.5
62.2
59.4
P
9.9
27.7
30.7
24.8
Table 4: The Execution Match (EXM) accuracies of the ablations on AmbiQT. Template Diversity helps with Join Ambiguity and Precomputed Aggregates, while Schema Diversity aids with Column/Table ambiguity.
shows, increasing beam width generally reduces coverage since the model is naturally biased to- wards one of the two alternatives, and greater beam width only serves let the model discover more vacuous variants of it. Using a larger num- ber of outputs does not help much, as attested to by Figure 4. Even a 3x increase in the number of outputs leads only to marginal improvements, except for Table Ambiguity.
6.4 Performance on Unambiguous Queries
Although our main focus is coverage under am- biguity, also evaluate proposal against the baseline T5-3B model on the dev split of SPIDER. We find that LogicalBeam doesn't just help the AmbiQT benchmark but also provides gains on conventional Text-to-SQL. benchmarks like SPI- DER where ambiguity limited. Table shows that LogicalBeam improves the top-5 Exact-Set and Execution Match accuracies on SPIDER by 2.3 % and 3.1 % over the bascline, respectively. As another example, we evaluate our method on the dev split of the challenging Kaggle DBQA (Lee et al., 2021) benchmark. We observe a drastic in- crease in the top-5 Exact-Set and Exccution Match accuracies, from 27.1 % and 26.5 % to 35.4% 4% and
Recent diversity-promoting decoding algo- rithms fail due to skewed token distribution alternatives Beam Search such as Nucleus and Typical sampling perform worse than beam search. Although these have demonstrated strong performance for tasks such as text summarization (Meister et al., 2023), they cannot promote meaningful diversity in se-
SOTA

---

Top-5 Exact Set Match
Top-5 Execution Accuracy
plan (prefix) could be replaced with any aspect of code snippet that we wish control. More gener- ally, since the underlying mechanism only involves the model being faithful to the prefix and has no manual components, we could do the same with al- most any Sequence-to tereenccett--quenee task (for example, political alignment in news summarization).
Method
SPIDER (dev split, %)
T5-3B
76.1
78.2
LogicalBeam 78,4
81.3
Kaggle DBQA (dev split, %)
T5-3B
27.1
26.5
LogicalBeam 35,4
35.4
LogicalBeam consistently improves perfor- mance both under ambiguity and in the absence of it, often by drastic margins. However, we would also like to highlight one failure mode we observed, that was also exhibited by other approaches. Con- sider a query .... table1 as t1 JOIN table2 as t2". On rare occasions, we observed that an identical query with t2 replaced by t3 (and t2 skipped) was also present in the choices. We be- lieve this indicates strong bias of the underlying model towards a particular template so much so that it prefers this weird (t1, t3) combination to introducing template diversity. The problem of debiasing the model makes for exciting future work. It is not unique to Semantic Parsing, and, we believe, deserves attention in its own right.
Table 5: The Exact-Set and Execution Match accuracies of LogicalBeam on two popolar Text-to-SQQ to-SQL datsets, SPIDER and Kaggle DBQA. Despite the datasets not exhibiting ambiguity, LogicalBeam delivers significant improvements over the T5-3B baseline.
35.4%, respectively. We conclude that Logical- Beam is useful across a wide range of Semantic Parsing tasks. Unlike earlier grammar-based gen- erators like SmBoP (Rubin and Berant, 2021) that require special decoder models, our approach can work within existing LM-based models.
6.5 Ablation study
LogicalBeam has three design decisions: (1) Use of two-step approach, (2) Counterfactual structural directives via plans, (3) Cemplate-guided schema diversity. We present an ablation study where we incrementally add these changes in Table 4. The first column ("Single Stage") generates an SQL directly with a prefix for structural diversity, dif- fering from LogicalBeam only in using single stage. It still uses plan enforcement and branch- ing control. We find that its coverage lags behind LogicalBeam, and by a large margin for T and P. The primary reason could be that template-guie guided decoding allows us to discard erroncous extensions at each decoding step. The second column ("Two Stages")) shows a simple two-stage method where we generate template without any counterfactual control, and use Beam Search to fill it in. This method decouples template and schema diversity, but cannot encourage either by itself. Forcing coun- terfactual diversity ("+Template Diversity")) boosts the coverage under Join Ambiguity and Precom- puted Aggregates. Finally, encouraging Schema Diversity via our Restricted Fill-In Algorithm (Log- icalBeam, the last column) significantly improves coverage for Column and Table Ambiguity.
Conclusion
In this work, we highlighted the lack of evaluation of Text-to-SQL models under ambiguity in con- temporary literature. To address this, we developed AmbiQT, novel benchmark with 3000+ challeng- ing examples that evaluates Text-to- Text-to-SQL models on four kinds of ambiguity. We demonstrated that cur- rent methods fall short of acceptable performance under ambiguity. Motivated by analyzing the er- rors of a T5-3B model on the SPIDER dataset, we developed two-step approach of generating and then filling template. To this end, we trained model predict the number of JOINs and selec- tions as a plan before the template, and controlled template diversity by enforcing appropriate plans. Beam Search was modified to enforce template ad- herence during in-filling. Our method aligns well with intuition and greatly improves a model's 's cov- erage under ambiguity, measured AmbiQT. It also delivers improvements in the absence of ambi- guity, on the SPIDER and Kaggle DBQA datasets. We hope our efforts inspire future work to study generation under ambiguity in more detail, both in the domain of Text-to-SOL. conversion and beyond.
6.6 Discussion
LogicalBeam is general and need not be confined to the world of Semantic Parsing. For instance, the

---

Limitations
bles: Exploring the power of tables on the web. VLDB.
In this work, we curated a benchmark of ambiguous queries by perturbing SPIDER, existing dataset. While we believe that our benchmark is good measure of performance under ambiguity, real-life databases may exhibit more numerous as well as varied forms of ambiguity. In addition, AmbiQT only consists of examples with questions in English. Ambiguity may manifest differently based on the choice of natural language, and a corresponding study should make for interesting future work.
Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang. Vit- torio Castelli, Patrick Ng. and Bing Xiang. 2023. Drspider: A diagnostic evaluation benchmark to- wards text-to-SQL robustness. In The Eleventh Inter- national Conference on Learning Representations.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained on code.
Due to the two-step approach, LogicalBeam in- curs higher number of decoding steps com- pared an end-to-end model. However, due to us- ing a lightweight Greedy Search for the first stage, the number of decoding steps of LogicalBeam falls not much beyond the baseline. Nevertheless, find- ing an optimal trade-off between decoding steps and coverage remains an intriguing challenge.
At the time of writing, ChatGPT and OpenAl Codex represent the most powerful publicly avail- able LLMS suitable for Text-to-SQL. conversion and are unable to exhibit sufficient diversity under ambiguity. Future versions or models may over- come this barrier.
Ethics Statement
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Zhao, Yanping Huang, Andrew Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling caarucctionfinetunet language models.
As we generate AmbiQT by perturbing SPIDER (Yu et al., 2018), it does not contain any identifying information. We do not foresee any broader ethical impacts of our work.
Acknowledgements
We thank Microsoft for sponsoring access to Azure OpenAl via the Accelerate Foundation Models Academic Research Initiative. We also extend our heartfelt gratitude to the Anonymous Reviewers who helped improve the paper with their insightful observations and suggestions.
Data Commons. 2009. Data commons. https:// datacommons org/ Accessed: 2022-05-25.
Ahmed Elgohary, Saghar Hosseini, and Ahmed Has- san Awadallah. 2020. Speak to your parser: Interac- tive text-to-SQL with natural language feedback. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2065- 2077, Online. Association for Computational Lin- guistics.
References
Samuel Arcadinho, David Aparício, Hugo Veiga, and António Alegria. 2022. T5ql: Taming language mod- els for sqi generation.
A Ahmed Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. NL-EDIT: Correcting semantic parse enrors through natural language interaction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Elgohary, Christopher Meek, Matthew
Abhiject Awasthi, Ashutosh Sathe, and Sunita Sarawagi. 2022. Diverse parallel data synthesis for cross- datahase adaptation of text-to-sql parsers.
Michael J. Cafarella, Alon Halevy, Yang Zhang, Daisy Zhe Wang, and Eugene Wu. 2008. Webta-

---

Language Technologies, pages 5599-5610, Online, Association for Computational Linguistics.
Gyubok Lee, Hyeonji Hwang, Scongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang. Minjoon Seo, Jong-Yeup Kim, and Edward Choi. 2022. Ehrsql: A practical text-to-sql benchmark for electronic health records. In Advances in Neural Information Process- ing Systems, volume 35, pages 15589-15601. Curran Associates, Inc.
Jenny Rose Finkel, Christopher D. Manning, and An- drew Y. Ng. 2006. Solving the problem of cascading errors: Approximate Bayesian inference for linguis- tic annotation pipelines. In Proceedings of the 2006 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1886666,666 Sydney, Australia. Association for Computational Linguistics.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In AAAI.
Han Fu, Chang Liu, Bin Wu, Feifei Li, Jian Tan, and Jianling Sun. 2023. Catsql: Towards real world natu- ral language to sqi applications. Proc. VLDB Endow., (56):1534-154,7
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promontt ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoit Sagot, and Rachel Bawden. 2022. Tackling ambi- guity with images: Improved multimodal machine translation and contrastive evaluation.
Jiwei Li and Dan Jurafsky. 2016. Mutual information and diverse decoding improve neural machine trams- lation.
Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL easier to infer from natural language specifications. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2030-2042, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Yihang Li, Shuichiro Shimizu, Weiqi Gu, Chenhui Chu, and Sadao Kurohashi. 2022. Visa: An ambigu- ous subtitles dataset for visual scene-awar machine translation.
Kevin Gimpel. Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. 2013. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natu- ral Language Processing, pages I 100-1111, Seattle, Washington, USA. Association for Computational Linguistics.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehensive evaluation of chatgpt's zero- shot text-to-sql capability.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally typical sampling.
Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, and Mirella Lapata. 2022. A well-composed text is half done! composition sampling for diverse conditional genera- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers). pages 1319-1339, Dublin, Ireland. Association for Computational Linguistics.
Moshe Hazoom, Vibhor Malik, and Ben Bogin. 2021. Text-to-SQL in the wild: A naturally-occurrigg dataset based on stack exchange data. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pages 77-87, Online. Association for Computational Linguistics.
John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3414- 3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
OpenAl. 2022. ChatGPT. https: //ppnaii com/ blog/ chattpt. [Online; last accessed May 2023].
Jonathan Pilault, Xavier Garcia, Arthur Bražinskas, and Orhan Firat. 2023. Interactive-chain-pee promapping.. Am- biguity resolution for crosslingual conditional gener- ation with interaction.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Rakesh Pimplikar and Sunita Sarawagi. 2012. Answer- ing table queries on the web using column keywords. In Proc. of the 38th Inl'l Conference on Very Large Databases (VIDDB).
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic evalu- ation of text-to-SQQ parsers. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- unn Long Pupers). pages 2261-2273 Online. As- sociation for Computational Linguistics.
Mohammadreza Pourreza and Davood Rafeii. 2023. Din-sql: Decomposed in-context learning of text- to-sql with self-correction.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits

---

of transfer learning with a unified text-to-text trans- former.
3174-3187, Brussels, Belgium. Association for Com- putational Linguistics.
Ohad Rubin and Jonathan Berant, 2021. SmBoP: Semi- autoregressive bottom-up semantic parsing. In Pro- ceedings of the 5th Workshop on Structured Predic- tion for NLP (SPNLP 2021), pages 12-21, Online. Association for Computational Linguistics.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong,
Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin. Sida Wang, Vic- tor Zhong. Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni. Ziyu Yao. Dragomir Radev, Caiming Xiong, Lingpeng Kong. Rui Zhang. Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.
Torsten Scholak, Raymond Li, Dzmitry Bahdanau, Harm de Vries, and Chris Pal. 2021a. DuoRAT: To- wards simpler to-L models. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1313-1322 1321, Online. Association for Computational Linguistics.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang, Zifan Li, James Ma. Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and eext-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 391 1-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021b. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
John M. Zelle and Raymond J. Mooney. 1996. Learn- ing to parse database queries using inductive logic programming. In Proceedings of the Thirteenth Na- tional Conference on Artificial Intelligence Volume
Noam Shazeer and Mitchell Stern. 2018, Adafactor: Adaptive learning rates with sublinear memory cost.
2, AAAI'9,, page 1050-1055. AAAT Press.
Mike Stonebraker, Daniel J. Abadi, Adam Batkin, Xue- dong Chen, Mitch Cherniack, Miguel Ferreira, Ed- mond Lau, Amerson Lin, Sam Madden, Elizabeth J. O' O'Neil, Patrick E. O' O'Neil, Alex Rasin, Nga Tran, and Stan Zdonik. 2019. C-store: a column-oriented DBMS. In Michael L. Brodie, editor, Making Databases Work: the Pragmatic Wisdom of Michael Stonebraker, pages 491-518. ACM / Morgan Clay- pool.
Lu Zeng, Sree Hari Krishnan Parthasarathi, and Dilek Hakkani-Tur 2022. N-best hypotheses reranking for text-to-sql systems.
Tianyi Zhang, Mina Lee, Lisa Li, Ende Shen, and Tat- sunori B. Hashimoto. 2022. Templm: Distilling lan- guage models into template-based generators.
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jason Eisner. 2022. Active programming by example with natural language prior.
Lappoon R. Tang and Raymond J. Mooney. 2000. Au- tomated construction of database interfaces: Integrat- ing statistical and relational learning for semantic parsing. In Proceedings of the 2000 Joint SIGDAT Conference on Empirical Methods in Natural Lan- guage Processing and Very Large Corpora: Held Conjunction with the 38th Annual Meeting of the Association for Computational Linguistics Volume 13, EMNLP '00, page 133-141, USA. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018 Seq2SQL: Generating structured queries from natural language using reinforcement learning.
A Computational Resources and Prompting Details
We highlight in this section the prompts we used for prompting ChatGPT, both for the synonyms of table/column names and for the Text-to-SQL con- version on AmbiQT. We also provide the prompts we used with OpenAl Codex, and furnish details of the computational resources we used, All details provided below are specified as of June 20, 2023.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578, Online. Association for Computational Linguistics.
Bing Wang. Yan Gao, Zhoujun Li, and Jian- Gan-G Lou. 2022. Know what don't 't know: Handling ambiguous and unanswerable questions for text-to-sql.
A.1 Computational Resources
All of our experiments were run on single NVIDIA A A100 GPU with 80GB of memory. We estimate the total GPU usage to have been roughly 500 GPU hours across training and inference. We
Sam Wiseman, Starr Shieber, and Alexander Rush. 2018. Learning neural templates for text generation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages

---

You are a helpful assistant that assists the user in deciding alternate names for + their tables in SQL database.
Listing I: The directive we use for asking ChatGPT produce table synonyms.
The database with database "[DB_ID]" currently has tables with the names [ TABLES_STRING]. Give me two alternate names for the table "[TABLE_NAME]] Print your output as a python list. Do not print any additional information, formatting, explanation, or notes.
Listing 2: The prompl we for asking ChatGPT produce table synonyms.
further estimate the cost of utilizing ChatGPT and OpenAl Codex be under 100$ in total.
asked ChatGPT to structure its output as JSON snippet, departure from the conventional prompt as in (Liu et al., 2023). This was motivated by our observation that ChatGPT would occasionally sneak comments or notes into its queries despite our best efforts. By asking it to produce the output in structured (ISON) format, it was much easier to detect errors and retry.
A.2 Synonyms (ChatGPT)
For column and table synonyms, we use one-shot prompting to indicate to ChatGPT the kind of trans- formation we desire.
For column synonyms, the overall directive and prompt are shown in Listings and respectively. The demonstrated example also follows the format the prompt. [DB_ID] is the database ID of the database having the column, and [TABLE_NAMG is the name of the table containing it. A comma- separated list of all database table names in quotes is filled into WTABLES_STRING]
We use the directive and prompt shown in List- ings and 6 respectively. The database ID and its schema go into [DB_ID] and [SCHEMA], respec- tively. The question is passed at the end in the placeholder [QUESTION]. Our demonstration for ChatGPT consists of using the question "Show the stadium name and the number of concerts in each stadium". and the output used for demonstration is shown in Listing 7.
Similarly, for table synonyms, the directive and prompt are shown in Listings 3 and respec- tively. In particular, [DB_ID] and [TABLE_NAME] are replaced with the database ID and table name. [COLUMN_NAMES) is comma-separated list of columns of the specified table. The demonstrated example also follows this format.
A.4 Texi-to-SQL (OpenAI Codex)
We found that asking Codex produce multiple SQLs in the same output did not have the desired effect, as it did not usually conform to the number of outputs or the format. Therefore, we instead prompt Codex multiple times with a temperature of 0.6 (as recommended by OpenAl to elicit cre- ativity) and a top-p of 0.7 to get its outputs. To this end, we found both zero and one-shot prompting ineffective in conveying to Codex the specific for- mat of the output (unlike ChatGPT). In contrast, we found that few (specifically, two) shot prompting to work much better, and therefore proceeded with that alternative. Our two demonstrations as well as the query prompt follow the format of Listing 8. The output formatting is simply the SQL query string inside curly braces. The two demonstrated examples are replicated in Listing 9.
We found that asking ChatGPT to structure its output as a JSON snippet saved us the trouble of sanitizing its outputs and separating it from any decoration (comments or explanation) it produced. It also made it easier to detect invalid outputs and retry.
A.3 Text-to-SQL (ChatGPT)
We prompted ChatGPT in a one-shot manner for evaluation on our benchmark. This was necessary as our benchmark is built by modifying SPIDER (Yu et al., 80. The queries are expected to be in a specific format in the spider dataset. In particular, the table aliases always t1, t2, ther, columns are never aliased, and only unquali- fied JOIN used and INNER JOINs, OUTER JOINS not used. Therefore, rather than do some ad-hoc post post-correction, we showed ChatGPT one example from the original SPIDER dev set. In addition, we
Fur-
B Alternate Prompts with ChatGPT
Before settling on our choice, we also experi- mented with existing prompts used by prior work (zero-shot, as opposed to our one-shot method).

---

You are helpful assistant that assists the user in deciding alternate names for 4 their tables' columns in an SQL database.
Listing 3: The directive we use for asking ChatGPT to produce column synonyms.
The database with database ID "[DB_ID]" has a table called "[TABLE_NAME]". This + table has columns with the following names: [COLUMN_NAMES]
Give me two alternate names for each column. Format your output as a json snippet + with keys corresponding to column names. Do not print any addi tional + information, formatting, explanation, notes.
Listing The prompt we use asking ChatGPT to produce column synonyms.
You are a helpful assistant that converts provided English questions to SQL queries with respect to provided schema.
Listing 5: The directive we use while prompting ChatGPT on our benchmark.
The schema for a database with Database ID [DB_ID] is: [SCHEMA]
Convert the following English question to the five most plausible SQL queries
+ compatible with the above schema.
Use simply the column name for selections in simple queries. For queries with joins use t1, t2, and
aliases for the tables, and use t1. t1.column, t2.
+ column, and so on for the column selections.
Structure your output as JSON snippet with single key "queries", mapping to
+ list of alternatives. Do not print any additional information, explanation, <> formatting, or notes.
Question: [QUESTION]
Listing 6: The prompt while prompting ChatGPT on benchmark.
{
"queries": [
"select t2. name, count(*) from concert t1 join stadium t2 on t1. stadium_id t2. stadium_id group by stadium_id""
"select t3. name, (* from concert as t1 join stadium as t2 on t1.
+ stadium_id t2. stadiumi join singer as t3 on ti.sirger_id t3. <> >ingerr_is id group by t1.stadium_id"
"select t3. name, count(*) from concert as t1 join stadium as t2 on t1. t2. + stadium_id stadtmmiu join singer_in_coneee
as t3 on t1. + concert_id t3. singerrid group by t1. t1.stadium_id""
"select t3. name, c*unt(*) from concert
t1 join stadium t2 t1. stadium_id t2. stadium_dd join singer_in_conee
t3 on t1.
+ concert_id t3. singer_id group by t1. st.dttuium_id
"select t1. name, count (*) from stadium as t1 join concert as t2 on t1.
4 stadium_id t2. stadium_iid group by t1. tt.stadium_id"
]
Listing 7: The demonstrated outputs for the one-shot example with the query "Show the stadium name and the number of concerts each stadium"..

---

# Use the schema links to generate the SQL query for the question
[SCHEMA]
Convert the following English question SQL queries compatible with the above + schema.
Use simply the column name for selections in simple queries. For queries with joins + use t1, t2 and so on as aliases for the tables, and use tl. to.uun,,, t2. column and so on for the column selections.
Question: [QUESTION]
Listing The format of both the demonstrated and query examples
Question List the official name and status of the city with the largest + population.
Query 1: SELECT official_name, status FROM city ORDER BY population DESC LIMIT Question 2: Show the stadium name and the number of concerts in each stadium.
Query 2: SELECT t1 name, rount(*) FROM stadium AS t1 JOIN concert AS t2 ON t1. stadium_id t2. stadiummm id GROUP BY t1.stadium_id
Listing 9: The examples used demonstrations for OpenAl Codex. The "Question" and "Query" indicators just for clarity, and the formatting is as per Listing 8.
###
Generate possible sqlite SQL queries ending with ';" for the question given + in triple backticks, with no explanation.
#*# Sqlite SQL tables, with their properties:
#
[CB_SCHEMA]
#
###
[QUESTION]
Listing 10: An alternate prompt used by prior work that tried.

---

Kind of Ambiguity
CGPT
Codex
RSQI.
F-T5-3B
T5-3B
T5-3B-k
T5-3B-p
T5-3B-T
Logical Beam
EitherInTopK (%)
C
43.4
51.5
49.4
58.6
57.6
52.3
51.9
48.2
65.8
T
41.1
54.3
30.2
58,9
56.5
47.6
47.3
41.9
66.8
J
67.4
82.6
68,1
87.5
85.4
82.3
82.3
79.5
89.2
P
51.5
80.2
57.4
65.4
78.2
72.3
74.3
69.3
81.2
Bothin (Coverage) (%)
19.8
9.4
10.5
12.2
0.0
27.7
28.9
11.6
4.7
5.7
0.0
39.9
15.3
41.0
0.0
22.6
24.3
2.8
1.7
0.0
57.3
P
7.9
23.8
7.9
3.0
16.8
4.0
2.0
0.0
22.8
Table 6: The Exact Set Match (EM) Accuracy of the compared systems.
Question
What are the names, countries, and ages for every singer in descending order of age?
Gold Queries
1. SELECT nationality, age FROM singer ORDER BY age DESC
2. SELECT citizenship, age FROM singer ORDER BY age DESC
Outputs of T5-3B with Nucleus Sampling
1. SELECT name, nationality, age FROM singer ORDER BY age DESC
2.
SELECT name, nationality, age FROM singer ORDER BY age DESC
3.
SELECT name, nationality, age FROM singer ORDER BY age DESC
4.
SELECT name, nationality, age FROM singer ORDER BY age DESC
5.
SELECT name. nationality, age FROM singer ORDER BY age DESC
Figure 5: Nucleus Sampling shows virtually diversity in top-5 outputs due to highly skewed probability
distribution leading to the same tokens being sampled each time.
Kind of Ambig- uity
Sccemm Diversity (LogiclBeeam)
However, as shown in Table 8, the results with this prompting method always lag behind those obtained with our main choice.
Single stage
Two stages
+Template Diversity
FitherlnTopp (%)
C
64.3
64.2
63.6
65.8
T
60.0
64.6
63.2
66.8
I
88.9
90.3
89.2
89.2
P
78.2
80.2
78.2
81.2
Kind of Ambiguity
ChatGPT (%) (Our Prompt)
ChatGPT (%) (Liu et al., 2023)
Both (Coverage) (%)
C
22.7
11.2
23.5
16.0
16.0
27.7
T
37.3
15.5
26.8
39,9
J
15.6
0.0
56.6
57.3
P
8.9
6.9
27.7
29.7
22.8
Table The results of ChatGPT with the prompt of (Liu et al., 2023) lags behind those obtained with our prompt. The better numbers are bolded.
Table 7: The Exact Match (EM) accuracies of the com- pared ablations on AmbiQT.
particular, tried the prompt used by (Liu al., 2023) evaluate ChatGPT on our benchmark with minor modifications (asking for five outputs instead of one). We showcase it in Listing 10.
Therefore, we decided to stick with our choice for the comparison in Subsection 6.3.

---

Example Template Outputs with Beam Search
1. SELECT column, AVG(column) FROM table GROUP BY column ORDER BY column
2. SELECT column, AVG(column) FROM table GROUP BY column ORDER BY column ASC
3. SELECT column, AVG(column) FROM table GROUP BY column ORDER BY column DESC
4. SELECT AVG(column), column FROM table GROUP BY column ORDER BY column
5. SELECT column, AVG(column) FROM table ORDER BY column
Figure 6: Vanilla Beam Search inadequate elicit meaningful template diversity. particular, diversity in the
number of JOINs selections is lacking.
C Exact Set Match Accuracies for the System Comparison and Ablation Study
E Examples of Templates
A template is generated by abstracting away col- umn names, table names, integer constants, and string literals from an SQL query. While these are only small fraction of the various features of the SQL query, they represent disproportion- ately large percentage of viable alternatives. for instance, a column name may be replaced by any of the numerous other ones to generate an (oth- erwise useless) alternative. By abstracting away these details, we avoid generating spurious alterna- tives by swapping these features with other ones at the template generation stage. In addition, by generating, e.g., column instead of t1. column for t1 name, avoid trivial alias swaps. Some exam- ples of templates for a few SQL queries are shown in Table 9, and the replacements carried out for each kind of abstraction are outlined in Table 10.
Here we report the Exact Match (EM) accuracies of our System Comparison and Ablation Study for both the EitheriaTOP and BothInTopK modes of evaluation.
The System Comparison on AmbiQT in terms of EM, and of the various decoding algorithms, when applied to T5-3B, are shown in Table 6. We observe that Exact Set Match (EM) follows the same trend Execution Match (EXM) under both headings, once again demonstrating the superior coverage of LogicalBeam.
The results of Ablation Study, in turn, shown in Table 7. The trend of EM also matches that of EXM here.
Inadequacy of Conventional Decoding Algorithms
SQL Query
Template
SELECT column FROM table
this section, we give some anecdotes to highlight the shortcomings of conventional decoding algo- rithms for our purposes. The example for the case of Beam Search when used with Text-to-SQL. model was given in the main material as Figure 2. We also give here anecdote of Nucleus Sam- pling in Figure Strikingly, all the outputs of Nucleus Sampling are the same. This was the case for many of the examples we manually appraised. Upon further investigation, we discovered that the model produced extremely skewed probability dis- tributions for its tokens - it was not uncommon for certain tokens to be assigned greater than 0.99 probability. This renders conventional decoding algorithms, including sampling-base methods, in- effective. Similarly, we found Beam Search (as well as sampling approaches) to be suboptimal for the case of Text-to- temmlllateee conversion, as Fig- ure 6 exemplifies.
SELECT name FROM singer
SELECT t1.bern. borr_state,, AVG(t2. age) FROM head AS t1 JOIN employee t2 ON t1.emp_id t2.empiid
SELECT column, AVG column) FROM table AS t1 JOIN table AS t2 ON column = column
SELECT column FROM table WHERE column number AND column string
SELECT last_name FROM head WHERE 56 AND first_name "John"
Table 9: Examples of templates.
Token type
Example
Abstraction
Column Table name
t1. name, age singer 2.3.5 "California"
column table
Number String
number string
Table 10: The abstractions in a template.

---

F Example Outputs From the Systems
We showcase example outputs from three chosen systems- our method, ChatGPT, and T3-3B on the various kinds of ambiguities of AmbiQTin Fig- ures 7 through 10. Note that the first two outputs of our approach are from T5-3B. We observe that approach is more consistent than the other two in incorporating all the possible queries.

---

(C)olumn Synonyms
Question
What are the names of documents that use templates with the code BK?
Gold Queries
1. SELECT (2.file_ name nami FROM templates AS t1 JOIN documents AS 12 ON t1.templae__1 jid 12.teml WHERE tltemplate. type code "BK"
2. SELECT 12.record FROM templates tl JOIN documents AS 12 ON templltte id 12.temp WHERE tl.tepllaee type code "BK"
LogicalBeam
1. SELECT (1.ille name FROM documents AS (1 JOIN templates AS 12 ON t1.temlate id 12.temllate id WHERE (2.temllate_teyyee type code = "BK"
2. SELECT t1 tl.file name, Ul.document. description FROM documents AS t1 JOIN templates AS ON 11.template id = 12.template id WHERE 12.template type_codee code = "BK"
3. SELECT rl.rocrrd name FROM documents AS tl JOIN templates AS 12 ON tl.temllate_id (2teepllee id WHERE 22tteeettteeeeeeeeeeeeeeeeeeeeee httppppllte_pp____eee_ code = "BK"
4. SELECT t1 [1.fflc_nanee FROM documents AS tl JOIN templates AS 12 ON templatee ji 12.template id pwwpppeeeeeeeeppppppppeeeeeeeeee = "BK" UNION SELECT tl.fil_nname FROM documents AS 11 JOIN templates AS 12 ON tl .tepllte_id 12.template_i WHERE 12.temllte__yyp caanplate__ype_cove copee_ddee---_eeeeeeeeeeee "BK"
WHERE L_.lemplate_yyee_ooee
5. SELECT il.ftle_name FROM documents AS (1 JOIN documents AS 12 ON tL.template id 12templatee ii WHERE 22temllate htppetypppoooooo code = "BK"
ChatGPT
1. SEL tl.rrelrdd_daaee name FROM templates t1 JOIN ref template_type _tplatte__ypessteeee AS ON tl.template_ng saapplatt_tyeeeeote eode
(2.template_type. __dde_oee JOIN documents AS ON tl.template_i (3.template_d id WHERE
12.template_yyp baclid__yee_______e 'BK'
2. SELECT (3record_ name FROM documents
13.emplate 12.tepllaee JOIN @emplate_types types AS ON tl.eellate httpppatettpeeeeeeeeeeeeeeeeeeee type_code 12.tmpl Stpplutee_yyyeeeeee type_code WHERE tltemplate type httpetyyppeeeeeeeeeeeeeeeeeeeeee code = 'BK'
JOIN templates
3. SELECT tl.rcorrd name FROM templates AS JOIN documents
t1.template
12.template id JOIN ref template template_types stppes AS t3 ON tlteeplltee btcpllate_ttye__oo___ type_ccceeeeeeeeeeeeeeeepttteeee t3.temllate htwpe/ppptttte_eccottt type_codeeeeeeeeeeeeeepeppooeeee WHERE 13.emplate_iyye_oodde
'BK'
4. SELECT 13.record name FROM documents AS 13 JOIN templates AS 12 ON 13.template id 12.template id WHERE @ammplate_typ__codee 'BK'
5. SELECT record name FROM templates AS tl JOIN documents AS 12 ON teeemplate id = te.platee id WHERE tl deemppaae_typp_cooeee 'BK'
T5-3B
1. SELECT al.file_nane name FROM documents AS tl JOIN templates AS t2 ON tl.template_ie iid 12.template id WHERE 12.template_yyye patplate__tye___deee_ = "BK"
2. SELECT tl tl.file_name name, tl.document_ description FROM documents AS tl JOIN templates AS ON t..eeplate__i 12teelle jd WHERE 12.template_yye tempplaee_uyee_oodee = "BK"
3. SELECT file, name FROM documents AS JOIN templates AS ON tl.template id 12.template id WHERE
12.teplate__yee Soauplatt_tyee_cdeee "BK"
4. SELECT tl.file
tl.record
FROM documents AS tl JOIN templates AS 12 ON ti.temlaaee id
12template id WHERE template type_typpeee code "BK"
5. SELECT dl.cuuumett. FROM documents AS JOIN templates AS 12 ON tl. t1.templtte id 12templatee id
WHERE 22.teplate_ttype tttpttt_ttt____eeeeeeee "BK"
Figure 7: Example outputs from our system, ChatGPT and T5-3B on the C portion of AmbiQT

---

(a)ble Synonyms
Question
Show the name and the release year of the song by the youngest singer.
Gold Queries
1. SELECT somg,name, rong_release__gg year FROM artist ORDER BY age LIMIT 1
2. SELECT name, song cong_release_yeee year FROM performer ORDER age LIMIT
LogicalBeam
1. SELECT song_name, name, song les___ellee__eeeeee___e_e__ year FROM artist ORDER BY age LIMIT 1
2. SELECT 12.song name, 12.song release__caa ycar FROM artist AS tl JOIN song_name name AS 12 ON tl.song_naam name
12ssonnnaame ORDER BY tl age LIMIT 1
3. SELECT song_name, wang_relese__vee
FROM performer ORDER BY age LIMIT
4. SELECT song_name, song_release_eee FROM performer ORDER BY age ase LIMIT
5. SELECT song_name, song oung_release_yeet FROM concert ORDER BY age LIMIT
ChatGPT
1. SELECT name, song_ renesseeease__eal FROM performer WHERE age (SELECT MIN(age) FROM performer)
LIMIT
2.
SELECT name, song_ release_yeaa FROM artist WHERE age = (SELECT MIN(age) FROM artist) LIMIT
3. SELECT nl.amm, slsong release year FROM performer
JOIN singer in concert 12 singerer id 12.singer_id ii WHERE tl age (SELECT MIN(age) FROM performer) LIMIT
4. SELECT namee tl.song release year FROM artist t1 JOIN singer. ioncert 12.sir WHERE age (SELECT MIN(age) FROM artist) LIMIT
ON tl.singer
5. SELECT nlme, tl.ong release year FROM performer JOIN singer in concert 12 singer id
-ingr id WHERE ale (SELECT MIN(age) FROM (SELECT FROM performer UNION SELECT FROM artist)) LIMIT
T5-3B
1. SELECT song name. song_releee cese__relese___eeeee year FROM artist ORDER BY age LIMIT 1
2. SELECT (2.song name, 12.song release year FROM artist AS t1 JOIN song name AS 12 ON tl.song name 12.song_name name ORDER BY 11 age L.MIT 1
3. SELECT t2.song name. t2,song release year FROM artist AS tl JOIN song name AS 12 ON tl.song name 12.song_nan name WHERF 11 .age (SELECT MIN(age) FROM artist)
4. SELECT song_ name tetg_relleee_eeeeeeeeee year FROM artist ORDER BY ase LIMIT
5. SELECT 12.song_namm name, lrconngrleasseeeeeeeeeee FROM artist AS tl JOIN performer AS 12 ON ti-sieger_id
12.singer_id ORDER BY age LIMIT
Figure Example outputs from our system, ChatGPT and T5-3B on T portion AmbiQT

---

oo)in Ambiguity
Question
Return the name, location, and district of all shops in descending order of numher of products.
Gold Queries
1. SEL name, location, district FROM shop ORDER BY number_products DESC
2. SELECT nnam,, location, 12.district FROM shop JOIN shop_district 12 tl.spoo_idde -ddpppid=-pp 12.shop_id ORDER tl.number. products DESC
LogicalBeam
1. SELECT name, location, district FROM shop ORDER BY number, products DESC
number_productss ECT name, location and district FROM shop ORDER BY number_ number_products products DESC
2.
3.
SELECT name, district, location FROM shop ORDER BY number_proddee __rrductssss DESC
4.
SELECT name, district FROM shop ORDER BY sumber_producte DESC
5.
SELECT tlaame, location,, 12.distriet FROM shop AS (I JOIN shop_ district AS (2 ON tl.shop jdd 12.shop_id id
ORDER BY tl.number srrnuberrproducts DESC
ChatGPT
1. SELECT name, Incation, district FROM shop JOIN siop_district shop.shop id shop_ district.shopp.iie ORDER BY number_prodlust DESC
2. SELECT name, location, district FROM shop_district JOIN shop shop.shop_it id shop_ citoppddtttcttshhop...
ORDER BY number_ppeee cuunbeerproduuts DESC
3. SELECT s.name, s.location, ad.district FROM shop AS s JOIN shop_ district AS s.shop _d = sd.shop
JOIN shop AS s2 sum(s2.number products) DESC
shop sd.shop_ia id GROUP BY s.name, sd.district, s.location ORDER
4. SELECT s.name, s.location, sd.district FROM shop district AS JOIN shop
s.shop sd.shop
JOIN shop AS s2 ON s2.shop_id sd.shop_id GROUP BY s.name, sd.district, s.location ORDER BY sum(s2.number_png ppanbbbr_rrrruts) DESC
5. SELECT name, location, district FROM shop_distrin JOIN shop ON shop.shop_id shop_ district.shop_id ORDER BY (SELECT count(*) FROM employee WHERE employee.city -shop.locatio) DESC
T5-3B
1. SELECT name. location, district FROM shop ORDER BY number products DESC
2.
SELFCC name, location and district FROM shop ORDER BY number number_products DESC
3. SELECT location, district FROM shop ORDER BY number_preee crrbuurrprooucts DESC
4.
SELECT name, location, district form shop ORDER BY number_productt DESC
5.
SELECT location, district of shop ORDER BY number_proddue DESC
Figure Example outputs from our system, ChatGPT and T5-3B on the J portion AmbiQT

---

(P)re-Computed Aggregates
Question
What the average and maximum capacity stadiums?
Gold Queries
1.
SELECT AVG(capacity). MAX(capacity) FROM stadium
2. SELECT avg capacity, max capacity FROM stadium caaiiitc
LogicalBeam
1. SEI avg_ cap_citiyyy, max capacity FROM stadium, capacity
2. SELECT avg_ capacity (SELECT AVG(capacity) FROM cadiuit_cpaciit)) and max_capacity (SELECT
MAX(capacity) FROM stadium)
3. SELECT AVGCcapaciyy MAX(capacity) FROM stadium
4. SELECT AVG(capacity). MAX(max MAX(max_capacctt) FROM stadium stadium_capacity
5. SELECT AVG(cupacity). MAX(capacity), stadium FROM stadium capacity
ChatGPT
1. SELECT AVG(capacity), MA MAX(capacity) FROM stadium
2. SELECT AVG(LI.cpacigg MAX(tI FROM stadium AS t1
3. SELECT AVG(11 ccpppcccc,, tl tl.max capacity FROM stadium capacity AS
4. SELECT AVG(L1 capacccc.. MAX((2.capacitt)) FROM stadium AS t1 JOIN stadium AS 12 on Uleaapcciy <=
12.capacity group by tl.capacity
5. SELECT AVG((2.capacity) MAX((2.capacitt) FROM concert AS tl JOIN stadium AS tl sl.stadium id 12.stadium id
T5-3B
1. SELECT avg avg_aaaagatio capacity, max capacity FROM stadium capacity
2. SELECT avg capcccactty (SELECT AVG(capacity) FROM ccdduittcaaactty) and max_capacity (SELECT MAX(capacity) PROM stadium)
3. SELECT avg capacity (SELECT AVGicapacity) FROM stadium, capuccty and max capacity (SELECT
max capacity FROM stadium capacity) FROM stadium
4. SELECT AVG(capacity). MAX(capacity) FROM stadium capacity
stadium_capacityy
5. SELECT cvg_capay (SELECT AVG(capacity) FROM aaddium_capacity and max_capacity (SELECT
MAX(capacity) FROM stadium capacity) stadium_capacitt)
Figure 10: Example outputs from our system, ChatGPT and T5-3B on the P portion of AmbiQT


Reviews:
[{"review_id": "umUYVR09Pe", "reviewer": "Reviewer_ePD9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and comprehensive insights! We would like to address some of the concerns raised below.\n\n- **Underlying principles** -- We highlight the underlying principles that motivated LogicalBeam in Section 5 (lines 303-315). Our goal was to carefully control structural diversity and lexical diversity and suppress vacuous diversity.  The type of structural diversity we chose to explore was guided by studying the typical sources of structural errors in the default beam-search on standard benchmarks (Not the new AmbiQT benchmark, to avoid bias).\n- **Design decisions** -- Our design decisions are as follows: (Lines 310-315, Lines 529-531) These decisions allowed us to realize the underlying design principles.\n  1. Use of a two-step approach: first step to generate diverse SQL structure as template, second step to fill in diverse schema elements \n  2. Diverse plan generation  achieved via conterfactually generated plans in the prefix (number of JOINs and SELECTs) upfront during template generation (Lines 335-340).\n  3. Diverse schema explorations achieved using a whitelist of tokens at schema positions during in-filling at the time of Beam Search for controlled diversity (Lines 360-371)\n- **Implementation details** -- (Sections 5.1, 5.2, 6.1, Algorithm 1 - Section F & Page 16) Both stages of LogicalBeam are fine-tuned versions of Flan T5-3B (max length 512), with an Adafactor optimizer (learning rate 1e-4 with the default HuggingFace parameters, and no decay). The models were trained for roughly 300 epochs, with checkpoint selection based on the highest template match & exact match, respectively (on the validation set, Beam Width 1 with vanilla Beam Search).\nOur datasets for the two stages are one-to-one maps of each example of SPIDER, with, e.g., the SQL query replaced by the corresponding template for the text-to-template model.\nWe use the HuggingFace `LogitsProcessor` for the Template-Infilling model, which allows us to modify logits at each decoding step. We set all the disallowed tokens’ logits to $-\\infty$ to implement the restricted beam search.\n\nWe will make all these points even clearer in future versions, and add more details in the appendix.\n- **Evaluation on other benchmarks/validation in a broader context** -- At the time of writing, the most prominent benchmark for text-to-SQL conversion was SPIDER. In the absence of ambiguity, our approach shows a gain of 3.1% (Execution Match) on SPIDER (section 6.3) compared to  the baseline beam-search inference.  \n\nWe provide additional results on the KaggleDBQA -- another challenging  public benchmark for text-to-SQL generation.\n\n| Method      | Top-1 Exact Match | Top-1 Execution Accuracy | Top-5 Exact Match | Top-5 Execution Accuracy |\n|-------------|------------------:|-------------------------:|------------------:|-------------------------:|\n| Beam (Baseline)        |              24.3 |                     20.3 |              27.1 |                     26.5 |\n| LogicalBeam |              24.3 |                     20.3 |              **35.4** |                     **35.4** |\n\nThe “Beam” baseline here is the default  inference method of converting text to SQL while “LogicalBeam” is our method. We find LogicalBeam offers consistent gains in the top-5 outputs increasing execution accuracy by almost 9 points. This further demonstrates that LogicalBeam, by exploring more logical diversity during inference,  is more capable of covering the correct SQL in the Top-5.\n- **Positioning w.r.t other template generation/slot-filling approaches** -- While prior approaches have utilized template-filling approaches to Semantic Parsing, we are, to our knowledge, the first to explicitly utilize this decoupling to encourage targeted types of diversity (such as that of templates via plan enforcement). We compare to one such method, RESDSQL, in section 6.2 and show significant improvements across the board. Further, as our ablation study in section 6.4 shows, using a two-step approach decouples the types of diversity, but taking advantage of this requires plan enforcement. In future versions, we will make this more explicit by adding a separate heading for template-based approaches in the Related Works section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The development of AmbiQT is the first benchmark specifically designed to test performance under four types of ambiguity. This will likely drive further exploration and innovation in handling ambiguity, a real-world challenge often overlooked in existing benchmarks.\n2. The paper's analysis of state-of-the-art methods, including prominent models like T5-3B, RESDSQL, OpenAI Codex, and ChatGPT, provides valuable insights into how they handle ambiguity. By highlighting the limitations of these models in representing ambiguity, the paper raises awareness and encourages the development of more robust solutions.\n3. LogicalBeam's consistent increase in generating all gold SQLs in the Top-5 choices by 1.5-2.5 times over the baselines on AmbiQT demonstrates its effectiveness and superiority, adding a novel and effective tool to the Text-to-SQL toolkit. Its design to specifically address ambiguity opens up new avenues for research and development in this challenging aspect of language understanding.", "reasons_to_reject": "1. The paper need to provide a thorough explanation of the LogicalBeam algorithm, including its underlying principles, design decisions, and implementation details.\n2. The evaluation of LogicalBeam is confined solely to the AmbiQT benchmark without extensive comparisons to other benchmarks, thus the results might be perceived as narrowly focused and lacking generalizability.\n3. While the creation of the AmbiQT benchmark is valuable, an overemphasis on this specific benchmark without a broader context or validation on other datasets could lead to questions about its applicability and relevance to other scenarios.\n4. While LogicalBeam is presented as a novel solution, there have been existing work of text2sql parsing in template generation and slot filling approaches. Proper positioning and differentiation from related work would be essential to mitigate this risk.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper tackles the challenge of ambiguity in natural language queries over real-life databases, a significant problem that existing benchmarks do not adequately capture. By developing the AmbiQT benchmark, consisting of over 3000 examples with lexical and/or structural ambiguity, the authors provide a more realistic testing ground for Text-to-SQL systems. The paper also introduces LogicalBeam, a new decoding algorithm designed to navigate the SQL logic space. The results show that LogicalBeam significantly outperforms state-of-the-art models in generating candidate SQLs and improves Exact and Execution Match Accuracies on the SPIDER dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w7lmu20lMq", "reviewer": "Reviewer_8D4B", "summary": "", "questions": "A. Could you provide more implementation details about the text-to-template model? For example, what is its base model? Is its training data the modified Spider training set? Any checkpoint selection methods? Any intrinsic evaluations of the quality of this text-to-template model?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for your thoughtful and considerate review. We address some of the raised concerns below.\n\n- **The categories of “Join Ambiguity” and “Precomputed Aggregates” should be called “diverse” instead of “ambiguous”** -- This is a thought-provoking point! In our work, we categorize any scenario where two semantically distinct queries are ambiguous, although it would be interesting to see if future benchmarks designate a separate category for diversity.\nFor the two cases above, although sometimes two columns in different tables have the same name, the two queries they give rise to are semantically different. This is because the two columns may contain different entries, and are treated as distinct by the evaluation script. We will mention this explicitly in future versions.\n- **Details of the training process** -- Both stages of LogicalBeam (template generation and infilling) are fine-tuned from Flan T5-3B (max length 512), with an Adafactor optimizer (learning rate 1e-4 with the default HuggingFace parameters, and no decay). The models were trained for roughly 300 epochs, with checkpoint selection based on the highest template match & exact match, respectively (on the validation set, Beam Width 1 with vanilla Beam Search).\nOur datasets for the two stages are one-to-one maps of each example of SPIDER, with, e.g., the SQL query replaced by the corresponding template for the text-to-template model. This replacement is done programmatically following Table 9.\nWe use the HuggingFace `LogitsProcessor` for the Template-Infilling model, which allows us to modify logits at each decoding step. We set all the disallowed tokens’ logits to $-\\infty$ to implement the restricted beam search. \n- **Intrinsic evaluation of the template model** -- On the dev split of SPIDER, the final checkpoint had **top-1** and **top-5** template match accuracies of **64.7%** and **80.0%**, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The new benchmark, AmbiQT, is a good starting point for the text-to-SQL parsing subfield to study the ambiguity problem.\n2. The experiments show that LogicalBeam decoding is a solid method for improving diversity and useful for text-to-SQL parsing in general.\n3. The paper is well-motivated by analysis of problems in existing text-to-SQL models and decoding methods.\n", "reasons_to_reject": "While I agree with the two lexical ambiguity categories in AmbiQT, I would rather call the structural categories “diversity”, instead of  “ambiguity”. To my understanding, the synthesized parse variants for “join ambiguity” and “precomputed aggregations” are semantically equivalent to the original ones. ", "questions_to_authors": "A. Could you provide more implementation details about the text-to-template model? For example, what is its base model? Is its training data the modified Spider training set? Any checkpoint selection methods? Any intrinsic evaluations of the quality of this text-to-template model?\n", "paper_topic_and_main_contributions": "The paper studies the natural language ambiguity problem in text-to-SQL parsing. It has two main contributions:\n1. A new benchmark, AmbiQT, synthesized by rules and ChatGPT. The new benchmark is challenging to existing text-to-SQL approaches.\n2. A new constrained decoding algorithm, LogicalBeam, that can improve token and structural diversity in text-to-SQL parsers.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "kvTMOLiePz", "reviewer": "Reviewer_XzKB", "summary": "", "questions": "- For the schema modification, was any validation done to ensure the ambiguous schemas are naturalistic?\n- Were there any attempts at end-to-end training for the skeleton generation + filling steps? Could this be beneficial?\n- Could the LogicalBeam approach transfer to other text-to-code generation tasks exhibiting ambiguity?\n- More analysis on where LogicalBeam falls short would be useful - are there certain ambiguity patterns it still cannot handle well?\n- Are there any larger implications of this work - e.g. for dialog systems that involve querying databases?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt appreciation to the reviewer for taking the time to leave a meticulous and discerning review. We would like to respond below to some of the points raised in the review.\n\n- **AmbiQT covers only certain predefined types of ambiguity** -- While designing AmbiQT, we tried to achieve two primary goals. First, we wanted to create a benchmark representing the most critical facets of ambiguity in real life. Second, we strived to compartmentalize the types of ambiguity as rigorously as possible, as we needed to develop programmatic means to evaluate a system’s performance on AmbiQT. As ambiguity is inherently a human/natural concept, it can be tricky to straddle the boundary between exhaustive coverage and automatic evaluation. We chose the specified categories as they allowed us to address both issues. We agree that real-life workloads probably exhibit even more ambiguity – however, our benchmark covers the most prominent types, as our error catalog corroborates (Section 5). We are excited to explore other types of ambiguity in future work.\n- **Shortcomings of LogicalBeam** -- One failure mode of existing methods also exhibited by LogicalBeam was as follows. Consider a query ` … table1 as T1 JOIN table2 as T2`. On rare occasions, we observed that an identical query with `T2` replaced by `T3` (and `T2` skipped) was also present in the choices. We believe this indicates a strong bias of the underlying model towards a particular template – so much so that it prefers this weird (`T1`, `T3`) combination to introducing template diversity. The problem of debiasing the baseline model makes for exciting future work. However, the problem of a model being biased toward its top choice w.r.t other reasonable choices is not unique to Semantic Parsing. We believe that it deserves attention in its own right.\n- **Can an end-to-end model work as well?** -- We implemented an end-to-end approach that predicts the plan (JOINs and SELECTs) followed by the SQL and have compared to it in the ablation study of Section 6.4 (called “Single Stage”). We found that the two-stage approach provides significant, sometimes substantial, improvements over the direct method. While this may seem surprising, it can be explained as follows: \n  * Since there is no template to guide the decoding in the direct approach, we can prune out illegal candidates only in retrospect when it becomes clear which positions the schema names were at, i.e., possibly many decoding steps into the future. \n  * With a two-step approach, we can prune wrong extensions earlier, allowing us to explore more fruitful options instead. A back-of-the-envelope calculation: assuming that the two-step approach admits $10\\%$ fewer tokens than the direct method at each decoding step. Over 64 tokens, this translates to a $(1/0.9)^{64} \\sim 1000$ times more effective search. \n  * In this regard, it is possible that a direct method that could somehow anticipate the positions with schema tokens could do even better, and constitutes future work.\n- **Experiments beyond SPIDER** -- We provide additional results on the KaggleDBQA, a challenging public benchmark for text-to-SQL generation.\n\n| Method      | Top-1 Exact Match | Top-1 Execution Accuracy | Top-5 Exact Match | Top-5 Execution Accuracy |\n|-------------|------------------:|-------------------------:|------------------:|-------------------------:|\n| Beam (Baseline)        |              24.3 |                     20.3 |              27.1 |                     26.5 |\n| LogicalBeam |              24.3 |                     20.3 |              **35.4** |                     **35.4** |\n\nThe “Beam” baseline here is the default  inference method of converting text to SQL while “LogicalBeam” is our method.. We find LogicalBeam offers consistent gains in the top-5 outputs increasing execution accuracy by almost 9 points. This further demonstrates that LogicalBeam, by exploring more logical diversity during inference,  is more capable of covering the correct SQL in the Top-5.\n- **Transfer to other text-to-code tasks/larger implications of the work** -- That’s a great point! Certain parts of LogicalBeam are generally beneficial. For instance, the plan of the number of JOINs and SELECTs could be replaced with any aspect of a code snippet that we wish to control. Since the underlying mechanism only involves the model being faithful to the prefix and has no manual parts, this applies to almost any sequence-to-sequence task (for example, political alignment in news summarization, or being general/specific in a question-answering task). As a more significant implication, we realized when working with LogicalBeam that Language Models (even the larger ones like ChatGPT)  often have a single thing they want to say. Despite there being multiple reasonable outputs, they prefer one strongly and continuously try to steer the decoding to that option even when the prefix aligns better with another. We believe that teaching models to acknowledge the validity of multiple outputs (through their log probabilities) is essential as their capability increases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thoroughly motivates the problem of ambiguity in text-to-SQL, which has been lacking in prior benchmarks. AmbiQT fills this gap through a principled data collection process.\n- Provides useful analysis and insights on limitations of current models in handling ambiguity.\n- Proposes a novel inference algorithm that meaningfully improves coverage over strong baselines. The two-step skeleton generation and filling approach is intuitive.\n- Results are comprehensive, with comparisons to multiple state-of-the-art models and ablations. Substantial gains are shown consistently.\n- Well-written and easy to follow overall. The problem is clearly defined and contributions are impactful.", "reasons_to_reject": "- While AmbiQT is a step forward, it still only covers certain predefined types of ambiguity on a modified version of an existing dataset. Testing on truly open-domain or cross-database ambiguity is an avenue for future work.\n- There could be more analysis digging into the errors of LogicalBeam to understand its limitations better.\n- The approach requires training separate models for the two steps, increasing training overhead. Exploring end-to-end training could be beneficial.\n- The gains on SPIDER are modest, so more analysis is needed on what types of ambiguity it helps with, and what AmbiQT adds beyond SPIDER.", "questions_to_authors": "- For the schema modification, was any validation done to ensure the ambiguous schemas are naturalistic?\n- Were there any attempts at end-to-end training for the skeleton generation + filling steps? Could this be beneficial?\n- Could the LogicalBeam approach transfer to other text-to-code generation tasks exhibiting ambiguity?\n- More analysis on where LogicalBeam falls short would be useful - are there certain ambiguity patterns it still cannot handle well?\n- Are there any larger implications of this work - e.g. for dialog systems that involve querying databases?", "paper_topic_and_main_contributions": "This paper tackles the problem of ambiguity in text-to-SQL semantic parsing, where a natural language query may map to multiple possible SQL queries. The main contributions are:\n\n- Development of AmbiQT, a new benchmark with over 3000 examples where each text maps to two valid SQL queries exhibiting different types of ambiguity (column, table, join, aggregates). \n- Analysis showing that current text-to-SQL models, even large pretrained ones, do not adequately cover ambiguous alternatives in their top-k outputs.\n- A new decoding algorithm, LogicalBeam, that substantially improves coverage of ambiguity by generating diverse SQL skeletons and filling them in a constrained manner.\n\nThe paper addresses an important problem in making text-to-SQL models more robust to real-world ambiguity, through both new data and methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work the authors address the ambiguity of mapping from natural language queries into SQL queries: a reasonable case in which the NL utterance can have more than one interpretation in terms of SQL queries. To study this phenomenon and measure to which extent current NL-to-SQL models can generate all viable interpretations within the top-k candidates, the authors create the new AmbiQT dataset in which two valid SQLs are written for each NL query. 

They proceed to show that, due to the token-level diversity of the beam search in SotA models,   these models fail when the two valid SQLs exhibit mutual structural diversity. To address this, the authors propose a meaningful a structural decoding algorithm that nagivates the space of SQL syntax, dubbed LogicalBeam. LogicalBeam is shown to perform comparably or better to SotA on standard NL-to-SQL evaluation datasets (on SPIDER, reported in the submission; and KaggleDBQA, additionally reported in the rebuttal) and drastically better on the new AmbiQT, i.e., in the setting for which it was primarily designed. 

All of the reviewers identify the new dataset and the algorithm as valuable contribution. The few concerns, primarily about the performance of LogicalBeam on the non-ambiguous NL-to-SQL datasets have been successfully resolved in the author-reviewer discussion.
Decision: Accept-Main