Semantic Decomposition of Question and SQL for Text-to-SQL Parsing
Authors: Ben Eyal
Moran Mahabi
Ophir Haroche
Amir Bachar
Michael Elhadad
Venue: EMNLP 2023 Findings (2023)
Keywords: semantic parsing
text-to-sql
question decomposition
compositionality
TL;DR: To improve compositional learning of text-to-SQL, we introduce QPL, an executable modular semantic SQL transformation. With Spider adapted to QPL we learn a Question Decomposer aligned with QPL, leading to effective parsing of complex queries.

Abstract: Text-to-SQL semantic parsing faces challenges in generalizing to cross-domain and complex queries. Recent research has employed a question decomposition strategy to enhance the parsing of complex SQL queries.However, this strategy encounters two major obstacles: (1) existing datasets lack question decomposition; 
(2) due to the syntactic complexity of SQL, most complex queries cannot be disentangled into sub-queries that can be readily recomposed.

To address these challenges, we propose a new modular Query Plan Language (QPL) that systematically decomposes SQL queries into simple and regular sub-queries. We develop a translator from SQL to QPL by leveraging analysis of SQL server query optimization plans, and we augment the Spider dataset with QPL programs. 

Experimental results demonstrate that the modular nature of QPL benefits existing semantic-parsing architectures, 
and training text-to-QPL parsers is more effective than text-to-SQL parsing for semantically equivalent queries.

The QPL approach offers two additional advantages: (1) QPL programs can be paraphrased as simple questions, which allows us to create a dataset of (complex question, decomposed questions). Training on this dataset, we obtain a Question Decomposer for data retrieval that is sensitive to database schemas. 
(2) QPL is more accessible to non-experts for complex queries, leading to more interpretable output from the semantic parser.

Semantic Decomposition of Question and SQL for Text-to-SQL Parsing
Ben Eyal and Amir Bachar and Ophir Haroche
Moran Mahabi and Michael Elhadad
Dept of Computer Science Ben Gurion University Beer Sheva, Israel
Abstract
and queries. State of the art text-to-SQL models show performance above 90% for easy queries, but fall to about 50% on complex ones (see Table I). This accuracy drop is particularly bothersome for non-experts, because they also find it difficult to verify whether complex SQL query corresponds to the intent behind the question they asked. In a study performed, we found that software engineers who are experts in SQL fail deter- mine whether a complex SQL query corresponds a question in about 66% of the cases (see Table 4), The risk of text-to-code models producing incor- rect results with confidence thus acute: complex SQL queries non- con-aligned with users intent will be hard to detect.
Text-to-SQL. semantic parsing faces challenges in generalizing to cross-domain and complex queries. Recent research has employed a ques- tion decomposition strategy enhance the pars- ing of complex SQL. queries. However, this strategy encounters two major obstacles: (1) ex- isting datasets lack question decomposition; (2) due to the syntactic complexity of SQL,, most complex queries cannot disentangled into sub-queries that can be readily recomposed.
To address these challenges, we propose a new modular Query Plan Language (QPL.) that sys- tematically decomposes SQL queries into sim- ple and regular sub-queries. We develop a trans- lator from SQL to QPL by leveraging analysis of SQL server query optimization plans, and we augment the Spider dataset with QPL programs. Experimental results demonstrate that the mod- ular nature QPL benefits existing semantic- parsing architectures, and training text-to-QPI. parsers is more effective than text-to-SQU. pars- ing for semantically equivalent queries. The QPL approach offers two additional advan- tages: (1) QPL programs can be paraphrased simple questions. which allows to create a dataset of (complex question, decomposed questions). Training on this dataset, we obtain a Question Decomposer for data retrieval that is sensitive to database schemas. (2) QPL is more accessible to non-experts for complex queries, leading more interpretable output from the semantic parser.
In this paper, we address the challenge of deal- ing with complex data retrieval questions through compositional approach. Based the success of the question decomposition approach for multi-hop question answering. recent work in semantic pars- ing has also investigated ways to deal with complex SQL queries with a Question Decomposition (QD) strategy. In another direction, previous attempts have focused on splitting complex SQL queries into spans (e.g., aggregation operators, join crite- ria, column selection) and generating each span separately.
In our approach, we start from semantic anal- ysis of the SQL query. We introduce a new inter- mediary language, which we call Query Plan Lan- guage (QPL) that is modular and decomposable. QPL can be directly executed on SQL. databases through direct translation to modular SQL Com- mon Table Expressions (CTEs). We design QPL to be both easier to learn with modern neural archi- tectures than SQL and casier to interpret by non- experts. The overall approach is illustrated in Fig. 1. We develop an automatic translation method from SQL QPI.. On the basis of the modular QPL program, we also learn how to generate a natural language decomposition of the original question.
1 Introduction
Querying and exploring complex relational data stores necessitate programming skills and domain- specific knowledge of the data. Text-to-SQL. se- mantic parsing allows non-expert programmers to formulate questions in natural language, convert the questions into SQL, and inspect the execution results. While recent progress has been remark- text-to-SQI.
able on this task, general models still face challenges on complex schemas

---

Question
What is the official language spoken in the country whose head of state is Beatrix?
Gold SQL
SELECT T2. Language FROM country AS T1
JOIN countrylanguage AS T2 ON T1.Code T2. CountryCode
WHERE T1. HeadOfftate
'Beatrix' AND T2. IsOfficial
'T'
Gold QPL
#1 = Scan Table [country] Predicate [HeadOfState = 'Beatrix']
Output [Code, HeadOfState]
#2 = Scan Table [countrylanguage]] Output [CountryCode, Language, IsOfficial]
#3 = Filter [#2] Predicate [IsOfficial 'T'J Output [CountryCode, Language]
#4 = Join [#1, #3] Predicate Co3ntryure #1. Code] Output [#3. Language
Computed Question Decomposition (QD)
#1 Scan the table country and retrieve the code and head of state of the country whose head of state is Beatrix
#2 Scan the table countrylanguage and retrieve the country codes, languages and if they're re official
#3 = Filter from #2 all the official languages and retrieve the country codes and languages
#4 = Join #1 and #3 based on the matching country codes and retrieve the language spoken in the country whose head of state is Beatrix
Predicted QDMR
#1 = return countries whose head of state is beatrix
#2 = return the official language spoken in the official language of #1
Example QPI. and Question Decomposition compared to the original SQL query from Spider and to the predicted QDMR question decomposition from (Wolfson al., 2020).
Figure I:
In contrast to generic QD methods such as QDMR (Wolfson et al., 2020), our decomposition takes into account the database schema which is refer- enced by the question and the semanties of the QPL operations.
SQL is a challenging target language for neural models. The search for target language that is easier to learn has been pursued in Text-to-SQU as well (Yu et al., 201 80; Guo et al., 2019; Gan et al., 2021). We can view QPL as another candi- date intermediary language, which in contrast previous attempts, does not rely on a syntactic anal- ysis of the SQL queries but rather on a semantic transformation into a simpler, more regular query language.
Previous research in semantic parsing has shown that the choice of the target language impacts a model's ability to learn to parse text into an accu- rate semantic representation. For instance, Guo et al. (2020) compared the performance of various architectures three question-answering datasets with targets converted Prolog. Lambda Calculus, FunQL., and SQL. They discovered that the same architectures produce lower accuracy (up to a 10% difference) when generating SQL, indicating that
In the rest of the paper, review recent work in text-to-SQL models that investigates interme- diary representations and question decomposition. We then present the Query Plan Language (QPL) we have designed and the conversion procedure

---

we have implemented to translate the existing large-scale Spider dataset into QPL. We then de- scribe how we exploit the semantic transforma- tion of SQL to QPL to derive a dataset of schema- dependent Question Decompositions. We finally present strategies that exploit the compositional na- ture of QPL to train models capable of predicting complex QPL query plans from natural language questions, and to decompose questions into data- retrieval oriented decompositions.
It employed to assess the generalization capabili- ties of text-to-SQL models to unseen schemas on complex queries. Recent datasets have increased the scale to more samples and more domains (Lan et al., 2023; Li et al., 2023). In this paper, we fo- cus on the Spider dataset for our experiments, as it enables comparison with many previous methods.
2.1 Architectures for Text-to-SQL
We investigate two main research questions: (RQ1) Is it easier to learn Text-to-QPL. modular, decomposable query language than to Iearn Text- to-SQL using Language Model based architectures; (RQ2) Can non-expert users interpret QPL outputs in an easier manner than they can for complex SQL queries.
Since the work of Dong and Lapata (2016), leading text-to-SQL. models have adopted attention- -aseed sequence to sequence architectures, translating the question and schema into well-formed SQL query. Pre-trained transformer models have improved per- formance as in many other NLP tasks, starting with BERT- models (Hwang et al., 2019; Lin et al., 2020) and up to larger LLMs, such as T5 (Raf- fel et al., 2020) in (Scholak et al., 2021), OpenAl CodeX (Chen et al., 2021) and GPT variants in (Ra- jkumar et al., 2022; Liu and Tan, 2023; Pourreza and Rafiei, 2023).
Our main contributions are (1) the definition of the QPL language together with automatic trans- lation from SQL to QPL and execution of QPL. on standard SQL servers; (2) the construction of the Spider-QPL dataset which enriches the Spi- der samples with validated QPL programs for Spi- der's dataset together with Question Decomposi- tions based on the QPL structure; (3) Text-to-QPL models to predict QPL. from a (Schema + Question) input that are competitive with state of the art Text- to-SQL models and perform better on complex queries; (4) a user experiment validating that non- expert users can detect incorrect complex queries better on QPL than on SQL.'
In addition to pre-trained transformer models, several task-specific improvements have been in- troduced: the encoding of the schema can be im- proved through effective representation learning Bogin al. (2019), and the attention mechanism of the sequence-to-sequene
model can be fine-tuned Wang et al. (2020a). On the decoding side, tech- niques that incorporate the syntactic structure of the SQL output have been proposed.
2 Previous Work
To make sure that models generate a sequence of tokens that obey SQL syntax, different ap- proaches have been proposed: in (Yin and Neu- big, 2017), instead of generating sequence of tokens, code-oriented models generate the abstract syntax tree (AST) of expressions of the target pro- gram. Scholak al. (2021) defined the constrained decoding method with PICARD. PICARD an independent module on top of a text-to-text auto- regressive model that uses an incremental parser to constrain the generated output to adhere to the target SQL grammar. Not only does this eliminate almost entirely invalid SQL queries, but the parser is also schema-aware thus reducing the number of semantically incorrect queries, e.g., selecting a non-existent column from specific table. We have adopted constrained decoding in our approach, by designing an incremental parser for QPL, and en- forcing the generation of syntactically valid plans.
Text-to-SQL parsing consists of mapping a ques- tion Q (xI, (n) and a database schema S = ttableicool] ,oll,), ).... tabler(colf col"" into a valid SQL query Y = (vi...., Mq). Per- formance metries include exact match (where the predicted query is compared to the expected one according to the overall SQL structure and within each field token by token) and execution match (where the predicted query executed on a database and results are compared).
Several large Text-to-SQL datasets have been created, some with single schema (Wang et al., 2020b), some with simple queries (Zhong et al., 2017). Notably, the Spider dataset (Yu et al., 200 database schemas with over 5K complex queries and 10K questions.
2018b) encompasses
All of the datasets and code are available on hptps:// gi thub. wtt.l/oooooboeeeeeeeee com/bgunlp/opl.

---

Difficulty
NatQQL + RAT-SQL
Din-SQL GPT-4
GPT3.5-turbo
Easy
91.6%
91.1%
87.7%
Medium
75.2%
79.8%
75.1%
Hard
65.5%
64.9%
72.5%
Extra Hard
51.8%
43.4%
53.9%
Overall
73.7%
74.2%
74.3%
Table 1: Spider Development Set baseline execution accuracy by difficulty level
2.2 Zero-shot and Few-shot LLM Methods
Table 1 which shows execution accuracy of lead- ing baseline models (Gan et al., 2021; Pourreza and Rafiei, 2023) per Spider difficulty level the development set. The GPT3.5-turbo results cor- respond to our own experiment using zero-shot prompt. Other methods have been used to demon- strate that current sequence to sequence methods suffer at compositional generalization, that is, sys- tems trained on simple queries, fail to generate com- plex queries, even though they know how to gen- erate the components of the complex query. This weakness is diagnosed by using challenging com- positional splits (Keysers al., 2019; Shaw et al., 2021; Gan al., 2022) over the training data.
With recent LLM progress, the multi-task capa- bilitics LLMs have been tested on text-to-SOL. zero-shot mode, a task-specific prompt is pre- fixed to a textual encoding of the schema and the question, and the LLM outputs an SQL query. Ra- jkumar et al. (2022); Liu et al. (2023), showed that OpenAI Codex achieves 67% execution accuracy. In evaluation, GPT-4 (as of May 2023) achieves about 74% execution accuracy under the same zero-shot prompting conditions.
Few-shot LLM prompting strategies have also been investigated: example selection strategies are reviewed in (Guo et al., 2023; Nan et al., 2023) and report about 85% execution accuracy when tested on Spider dev or 7K examples from the Spi- der training set. Pourreza and Rafiei (2023); Liu and Tan (2023) are top performers on Spider with the GPT4-based DIN-SQL. They use multi-step prompting strategics with query decomposition. Few-shot LLM prompting methods close the gap and even outperform specialized Text-to-SQL mod- els with about 85% execution match vs. 80% for 3B parameters specialized models on the Spider test set, without requiring any fine-tuning train- ing. In this paper, we focus on the hardest cases of queries, which remain challenging both in SQL and in QPL (with execution accuracy at about 60% in the best cases). We also note that OpenAI-based models are problematic as baselines, since they cannot be reproduced reliably.
One of the reasons for such failure to generalize to complex queries relates to the gap between the syntactic structure of natural language questions and the target SQL queries. This has motivated a thread of work attempting to generate simpli- fied or more generalizable logical forms than exe- cutable SQL queries. These attempts motivated by empirical results other semantic parsing for- malisms that showed that adequate syntax of the logical form can make learning more successful (Guo et al., 2020; Herzig and Berant, 2021).
Most notable attempts include SyntaxSQLNet (Yu al., 2018a), SemQL (Guo et al., 2019) and NatSQL (Gan et al., 2021). NaIQL aims at re- ducing the gap between questions and queries. It introduces a simplified syntax for SQL from which the original SQL can be recovered. Figure 2 illus- trates how this simplified syntax is aligned with spans of the question.
2.3 Intermediary Target Representations
Most text-to-SQL systems suffer from severe drop in performance for complex queries, re- ported for example in DIN-SQL results where the drop in execution accuracy between simple queries and hard queries is from about 85% to 55% (see also (Lee, 2019)). We demonstrate this drop in
Our work is directly related to this thread. Our approach in designing QPI. is different from Nat- SQL, in that we do not follow SQL syntax nor attempt to mimic the syntax of natural language. Instead, we apply semantic transformation on the SQL query, and obtain a compositional regular query language, where all the nodes are simple ex- ecutable operators which feed into other nodes
II is most likely that the Spider dataset was part of the training material processed by GPT-x models.

---

Question
values (Wolfson al., 2022). LLMs have also been found effective as generic question decomposers in Chain of Thought (CoT) methods (Wei et al., 2022: Chen et al., 2022; Wang et al., 2023). In this work, we compare our own Question Decomposi- tion method with the QDMR model (Wolfson et al., 2022).
What type of pet is the youngest animal, and how much does it weigh?
SQL
SELECT PetType Weight FROM Pets ORDER BY Pgt_Age LIMIT 1
Decomposing Queries into QPL
Spider-SS Decomposition
SubSentence: What type of pet NatSQL: SELECT Pettype SubSentence: is the youngest animal NaISQL: ORDER BY Pets. Pet_Ag LIMIT SubSentence: and how much does it weigh? NatSQL: SELECT Pets. Weight
3.1 Query Plan Language Dataset Conversion
We design Query Plan Language (QPL) as a modu- lar dataflow language that encodes the semantics of SQL queries. We take inspiration in our semantic transformation from SQL to QPL from the def- inition of the execution plans used internally by SQL optimizers, e.g., (Selinger et al., 1979). We automatically convert the original Spider dataset into a version that includes QPL expressions for all the training and development parts of Spider. The detailed syntax of QPL is shown in $ A.4.
Figure 2: NatSQL and Question Decomposition in Spider-SS (Gan al., 2022)
a data-flow graph according to the execution plan of the SQL query. Our method does not aim at simplifying the mapping of a single question to a whole query, but instead at decomposing a question into a tree of simpler questions, which can then be mapped to simple queries. The design of QPL. vs. SQL adopts the same objectives as those defined in KoPL SparQL in (Cao al., 2022) in the setting of QA over Knowledge Graphs.
QPL is a hierarchical representation for execu- tion plans. It is tree of operations in which the leaves are table reading nodes (Scan nodes), and the inner nodes are either unary operations (such Aggregate and Filter) or binary operations (such as Join and Intersect). Nodes have arguments, such as the table to scan in a Scan node, or the join predicate of a Join node.
An important distinction between QPL plans and SQL queries is that every QPL sub-plan is valid executable operator, which returns a stream of data tuples. For example, Fig. 1 shows an execution plan with steps and depth 2. The steps are: the two Scan leaves, the Filter sub-plan, and the Join sub-plan, which the root of the overall plan. We automatically convert SQL queries into se- mantically equivalent QPL plans by reusing the ex- ecution plans produced by Microsoft SQL. Server 2019 query optimizer (Fritchey, 2018). QPL is a high-level abstraction of the physical execution plan produced (which includes data and index statistics). In QPL. syntax, we reduced the num- ber of operators to the 9 operators listed in Table 2. We also design the operators to be context free, i.e., all operators take as input streams of tuples and out- put stream of tuples, and the output of an operator only depends on its inputs. We experiment with different syntactic realizations of QPL expressions,
2.4 Question Decomposition Approaches
Our approach is also inspired by work attempting to solve complex QA and semantic parsing using question decomposition strategy (Perez al., 2020; Fu et al., 2021; Saparina and Osokin, 2021; Wolf- son et al., 2022; Yang et al., 2022; Deng et al., 2022b; Zhao et al., 2022; Niu et al., 2023). In this approach, the natural language question is decom- posed into a chain of sub-steps, which has been pop- QA
ular in the context of Soowledge-Grap--aaaee with multi-hop questions (Min et al., 2019; Zhang al., 2019). Recent work attempts to decompose the questions into trees (Huang al., 2023), which yields explainable answers (Zhang et al., 2023).
In this approach, the question decomposer is sometimes learned in joint- moint-manner to optimize the performance of an end to end model (Ye et al., 2023); also be derived from syntactic anal- ysis of complex questions (Deng et al., 2022a): or specialized pre-training of decompositions using distant supervision from comparable texts (Zhou et al., 2022); or weak supervision from execution
"This is in contrast to SQL execution plan operators such as Nested-1 Lops where the two children nodes tightly depend on cach other.

---

Operator
Description
Scan
Scan all rows in a table with optional filtering predicate
Aggregate
Aggregate a stream of tuples using a grouping criterion into stream of groups
Filter
Remove tuples from a stream that do not match a predicate
Sort
Sort stream according to sorting expression
TopSort
Select the top-K tuples from stream according to a sorting expression
Join
Perform a logical join operation between two streams based on join condition
Except
Compute the set difference between two streams of tuples
Intersect
Compute the set intersection between two streams of tuples
Union
Compute the set union between two streams of tuples
Table 2: Description of QPL Operators
Optimizer Exec. Plan
SQL Query
QPL
CTE
Results
Figure 3: QPL generation process: the dataset SQL expressions are run through the query optimizer, which is then converted into QPI.. QPI. expressions are converted modular CTE SQL. programs, which can executed. We verify that the execution results match those of the original SQL queries.
and elected the version where steps are numbered and ordered bottom-up, corresponding roughly to the execution order of the steps. We validate that the result sets returned by the converted QPL plans are equivalent to those of the original Spider SQL queries. We thus enrich the Spider training and de- velopment with semantically equivalent QPL programs as shown in Fig. 3.
In order to train a text-to-QPL model, we fine- tune Flan-T5-XL. (Chung et al., 2022) (3B param- eters) on 6,509 training examples. Each example contains question, schema information and the gold computed QPL. The input to the model is the same as in Shaw et al. (2021), i.e., Question | Schema Name Table1 Col11, Col12, Table2 Co121, Col22, We also exper- iment with rich schema encoding, adding type, key and value information as described in $A.2. We train the model for 15 full epochs and choose the model with the best exccution accuracy on the de- velopment set. Execution accuracy is calculated by generating a QPL prediction, converting it to Com- Table Expression format (CTE) (see example in Fig. 4), rumning the CTE in the database and comparing the predicted result sets of the predicted and gold CTES, Final evaluation of the model uses the PICARD (Scholak et al., 2021) decoder with a parser we developed for QPL syntax. This con- strained decoding method ensures that the gener- ated QPL programs are syntactically valid.
3.2 Text-to-QPL Model
Original SQL
SELECT template_id, count(*) FROM Documents
GROUP BY template_id
CTE
WITH
Scan_1 AS (
SELECT Template_ID FROM Documents
)
Aggregate_2 2 AS (
SELECT COUNT(*) AS Count, Template_ID FROM Scan_1
GROUP BY Template_ID
Question Decomposition
SELECT FROM Aggregate_2
We use the QPL plans automatically computed from SQL queries in the dataset to derive a set
Figure 4: SQL query and its equivalent CTE

---

question decompositions (QD) that are grounded in
4.3 Question Decomposer Model
the QPL steps, as shown in Fig. 1. We investigate three usages of this QD method: (1) QPL -> QD: we learn how to generate a QD given a QPL plan; this is useful at inference time, to present the pre- dicted QPL to non-expert users,
Given the dataset of <Q, QD> obtained above we train a QPL Question Decomposer which learns predict QD in our format given a question and a schema description: Q+Schema + QD. We fine- tunc a Flan-T5-XL model for this task, using the same schema encoding as for the Q + Schema -> QPL model shown in $3.2.
more readable form of the plan; (2) Q > QD we train question decomposer on the Spider- CPaler-PLLLa dataset for which we collect set of validated automatically gener- ated QDs; (3) Q + QD- + QPL we finally inves- tigate a Text-to-QPL LPL predictor model which given a question, firstly generates a corresponding QD, and then predicts a QPL plan based on (Q+QD).
4.4 Q+QD to QPL Prediction
We train a Flan-T5-XL model under the same con- ditions as previous models on QQ,QD, QP) to predict QPL given the QD computed by our ques- tion decomposer.
4.1
QPL to QD
We use the OpenAI 3 S-tuubo--000 model to generate QD given QPL plan. We prepared few-shot prompt that includes detailed descrip- tion of the QPL language syntax and six examples that cover all the QPL operators that we prepared manually (see {A.3).
5 Experiments and Results
5.1 Text-to-QPL Prediction
We present results the Spider development set in Table 3. We compare our models to T5-3B with PICARD (Scholak et al., 2021), as is the closest model to ours in terms of number of pa- rameters, architecture, and decoding strategy. To make the comparison as close as possible, we re- train model <Q + SQL> using the same base model Flan-T5-XI. as we use for our <Q -> QPI.> model. We also compare two schema encoding methods: Simple Schema Encoding only provides the list of table names and column names for each table; Rich Schema Encoding provides for each col- umn additional information: simplified type (same types as used in Spider's dataset text, number, date, other), keys (primary and foreign keys) and values (see S. for details). We see that at every difficulty level (except "Easy" for Simple Schema Encoding), Q QPL model improves the baseline. The same is true compared to the other models in Table 1.5 All other things being equal, this experiment indicates that easier to learn QPL as a target language than SQL (RQ1).
We validated manually 50 pairs (QPL, QD) gen- erated using this method and found them to be re- liable, varied and fluent. In addition, we designed an automatic metric to verify that the generated QDs are well aligned with the source QPL plan: (1) we verify that the number of steps in QD is the same that in the source QPL: (2) we iden- tify the leaf Scan instructions in the QD and verify that they are aligned with the corresponding QPL Scan operations. To this end, we use a fuzzy string matching method to identify the name of the table to be scanned in the QD instruction. The QPL-QD alignment score combines the distance between the length of QD and QPI. and the IoU (intersection over union) measure of the set of Scan operations.
4.2 Dataset Preparation
Using the QPL -> QD generator, we further enrich the Spider- QPL dataset with a computed QD field for each sample. For the sake of comparison, we also compute the predicted QDMR decomposition of the question (Wolfson et al., 2020) using the Question Decomposer from (Wolfson et al., 2
On overall accuracy, the direct Q -> QPL model achieves respectable 77.4% without database content and 83.8% with database content. Our model notably achieves highest exccution accuracy on Hard and Extra-Hard queries across existing tne-tuned and LLM- based models (70.0% across Hard+Extra-Hann Hard with database content).
We obtain for each example a tuple: <Schema, Question, SQL, QPL, QD, QDMR>. We obtained 1, 1,007 valid QDs (QPL-QD alignment score of 1.0) in the Spider Dev Set (out of 1,034) and 6,285 out of 6,509 in the Training Set with a valid QPL.
'The Q - SQL baseline we show reaches 82% vs. 79% reported in Scholak et al. (2021) as we used more complete schema encoding and the instruction fine-tuned X- model as opposed to the hase T5-3B model.
*Wo used the decomposer model pub- lished in httpp:///gtthue cwm/ tome commewwolgiii thub/ question question-decompsiieee dewcompoiiiion-ooooooooo sql.

---

Spider Difficulty
+ QPL
Q+QD - QPL
Q SQL
Support
Simple Schema Encoding
Easy
87.5%
84.7%
91.9%
248
Medium
84.3%
72.2%
76.9%
446
Hard
66.7%
62.0%
64.9%
174
Extra Hard
54.8%
45.1%
44.6%
166
Overall
77.4%
69.1%
73.3%
1034
Rich Schema Encoding
Easy
93.5%
91.5%
248
Medium
89.0%
88.3%
446
Hard
74.7%
69.5%
174
Extra Hard
65.1% 1
63.9%
166
Overall
83.8 %
82.0%
1034
Table 3: Accuracy on Spider Development Set by difficulty level with Simple Schema Encoding (table names and column names) and Rich Schema Encoding (column types, keys, values).
Type
Gold label
Time
Correct
(6 easy, 18 medium, 15 hard, 11 extra hard). This diversity is interesting, because for hard and extra- hard cases, execution accuracy remains low (55%- 74%). Showing multiple candidates from different models may be viable strategy to indicate the lack of confidence the model has on these queries.
QPL
Incorrect query Correct query
89%
53%
100%
79%
SQL
Incorrect query Correct query
102%
50%
101% %
25%
34%
Avg time
QPL SQL
123s
5.2 Interpretability User Experiment
132s
Table 4: User experiment: 20 (question, query) pairs shown to 4 users half in QPL and half in SQL, half are table reports how long
In order to probe whether QPL is easier to interpret by non-expert SQL users, organized user ex- periment. We selected sample of 22 queries of complexity Hard and Extra-Hard. We collected pre- dicted SQL queries and QPL plans for the queries, with half correct (producing the expected output), and half incorrect.
correct and half incorrect. assessed average each query, and how often they were right in assessing the correctness of the query.
The <Q QD > QPL model is inferior to the direct Q + QPL model (69.1% % to 77.4% with simple schema encoding). We verified that this is due to the lower quality of the QD produced by our question decomposer. On Oracle QD, the model produces about 83% accuracy without database content. Table 8 confirms that the model accuracy level increases when the QD-QPL alignment score increases. This indicates that the development of a more robust question decomposer grounded in query decomposition for training signal has the potential to improve semantic parsing performance.
Four volunteer software engineers with five years of experience participated in the experiment. We asked them to determine whether query either SQL or QPL corresponded to the intent of the natural language question. The participants were each exposed to half cases in QPL and half in SQL, half correct and half incorrect. We measured the time it took for each participant to make decision for cach case. Results reported in Table 4. They indicate that the participants were correct about QPL in 67% of the cases 34% of the SQL cases, supporting the hypothesis that validating the alignment between a question and query is casier with QPL than with SQL (p < 0.06) (RQ2).
In addition, we find that the <Q+QD QPL.> model produces correct answers that were not com- puted by the direct <Q + QPL> model 50 cases

---

Error Type
Error
Err. %
Explanation
Wrong aggregate
35
21%
Error in Aggregate (sum, avg, count, max, min)
Join
31
19%
Wrong join (e.g., not on Primary/Foreign key)
Wrong column
17
12%
Output does not include the right columns
Missing filter
17
10%
Filter stage is missing
Wrong constant
15
9%
Compare with wrong constant (e.g., isOfficial = 'Y' TT)
Wrong predicate
12
7%
Error in selection predicate (e.g.. > instead of <)
Lost
12
7%
Predicted QPL completely wrong
Typing
4%
Compare with constant of wrong type (e.g., age = 'old'' vs. 20)
Extremum
2%
Error in selecting top value (min vs. max for example)
Intersect
4
2%
Error in Intersect operation
Wrong structure
3
2%
QPL plan is not connected tree
Syntax issue
3
2%
Predicted QPL is not syntactically valid
Except
2
1%
Error in Except operation
Distinct
1
1%
Missing distinct flag
Wrong table
1
1%
Refers the wrong table in the schema
Grand Total
167
Table 5: Error Types: Breakdown of errors by error types
6 Error Analysis
automatically translate SQL queries into QPL and to execute QPL plans by translating them CTE SQL statements. We also compiled Spider-QPL, a version of Spider which includes QPL. and Ques- tion Decomposition for all examples.
The most challenging error type related to queries that involve aggregate operations (group by and aggregated values such as count, sum or max). Table 5 shows the breakdown of errors by error types for the Q >> QPL model with Rich Schema Encoding. We hypothesize that errors related to Join operations could be reduced by exploiting more expressive description of the schema struc- ture and applying either a post-processing critique of the generated QPL or enforcing stricter con- straints the QPL. Picard parser. Similarly, a more detailed description of the content of encoded columns could improve the Wrong Consiani type.
Our experiments indicate that QPL is easier to learn using fine-tuned LLMs (our text-to-QPL model achieves SOTA results on fine-tuned mod- els Spider dev set without db values, especially on hard and extra-hard queries); and easier to in- terpret by users than SQL for complex queries. On the basis of the computed QPL plans, we de- rived a new form of Question Decomposition and trained question decomposer that is sensitive to the target database schema, in contrast to existing generic question decomposers. Given predicted QPL plan, we derive readable QD that in- creases the interpretability of the predicted plan.
The Spider development includes 20 different schemas. Table shows the break down of errors per schema. We observe that schemas account for over 70% of the errors. These schemas do not follow best practices in data modeling: keys are not declared, column naming is inconsistent, and value encoding in some columns is non standard (e.g., of 'T''F' for boolean values).
In future work, we plan further exploit the modularity of QPL plans for data augmentation and to explore multi-step inference techniques. We have started experimenting with an auto-regressive model which predicts QPL plans line by line. Our error analysis indicates that enhancing the model to further take advantage of the database values and foreign-keys has the potential increase the robustness of this multi-step approach. We are also exploring whether users can provide interactive feedback on predicted QD as a way to guide QPL prediction.
7 Conclusion
We presented method improve compositional learning of complex text-to-SQL models based on QPI., a new executable and modular intermediary language that derived from SQL through seman- tic transformation. We provide software tools to

---

Limitations
Acknowledgements
All models mentioned in this paper were trained on one NVIDIA H100 GPU with 80GB RAM for 10 epochs, totaling around 8 hours of training per model at the cost of US$2 per GPU hour.
We thank the Frankel Center for Computer Science at Ben Gurion University for their support.
The models were tested on the Spider develop- ment set, which only has QPLs of up to 13 lines; method has not been tested on longer QPLs. During training, we did not use schema infor- mation such as the primary-foreign key relation- ships and column types, nor did we use the ac- tual databases' content. In this regard, our models might output incorrect Join predicates (due to lack of primary-foreign key information), or incorrect Scan predicates (due to lack of database content).
References
Ben Bogin, Jonathan Berant, and Matt Gardner. 2019. Representing schema structure with graph neural net- works for text-to-SQL parsing. In Proceedings of the 57th Anmual Meering of the Association for Computa- tional Linguistics, pages 4560-445 45600-4555550 Florence, Italy. Association for Computational Linguistics.
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang. Lei Hou, Juanzi Li, Bin He, and Han- wang Zhang. 2022. KQA pro: dataset with explicit compositional programs for complex question an- swering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6101 6119, Dublin, Ireland. Association for Compu- tational Linguistics.
We acknowledge certain limitations arising from the evaluation conducted on the Spider develop- ment set. These limitations include:
1. A total of 49 queries yield an empty result set on their corresponding databases. Conse- quently, inaccurately predicted QPLs could generate the same "result" as the gold query while bearing significant semantic differences. In the SPi. dataset, we have inserted additional data so that none of the queries re- turn an empty result set in the development set.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter. Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder. Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained code.
2. As many 187 queries employ the LIMIT function. This can lead complications in the presence of "ties"; for instance, when considering the query SELECT grade FROM students ORDER BY grade DESC LIMIT 1, the returned row becomes arbitrary if more than one student shares the highest grade.
Ethics Statement
The use of text-to-code applications inherently car- ries risks, especially when users are unable to con- firm the accuracy of the generated code. This issue is particularly pronounced for intricate code, such as complex SQL queries. To mitigate this risk, we introduce a more transparent target language. How- ever, our limited-scale user study reveals that, even when utilizing this more interpretable language, software engineers struggle to detect misaligned queries in more than 30% of instances. This occurs even for queries with moderate complexity (QPL length of to 7). More work on interpretability of generated code is warranted before deploying such tools.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li. Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction- arXiv preprint arXiv:2200. 1416.
language models.
Zhenyun Deng, Yonghua Zhu. Yang Chen, M. Witbrock. and Patricia J. Riddle. 2022a. Interpretable amr- based question decomposition for multi-hop ques- tion answering. International Joint Conference on Artificial Intelligence.

---

Zhenyun Deng, Yonghua Zhu, Yang Chen, Michael Wit- brock, and Patricia Riddle. 2022b. Interpretable amr- based question decomposition for multi-hop ques- tion answering. In Proceedings of the Thirty-First International Join! Conference on Artificial Intel- ligence, IJCAI-22, pages 4093-4099. International Joint Conferences on Artificial Intelligence Organi- zation. Main Track.
Jonathan Herzig and Jonathan Berant. 2021. Span- based semantic parsing for compositional general- ization. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 908-921, Online. Association for Computa- tional Linguistics.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 33-43, Berlin, Germany, Association for Computa- tional Linguistics.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Xiang Huang. Sitao Cheng, Yiheng Shu, Yuheng Bao, and Yuzhong Qu. 2023. Question decomposition tree for answering complex questions over knowledge bases.
Grant Fritchey. 2018. SQL Server Execution Plans: Third Edition. Red Gate Books.
Wonseok Hwang. Jinyeung Yim, Seunghyun Park, and Minjaon Seo. 2019. A comprehensive exploration wikisql with table-aware word contextualization. CoRR, abs/1902.01069.
Ruiliu Fu, Han Wang. Xuejun Zhang, Jun Zhou, and Yonghong Yan. 2021. Decomposing complex ques- tions makes multi-hop QA casier and more inter- pretable. In Findings of the Association for Compu- national Linguistics: EMNLP 2021, pages 169-180, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin. Nikola Momchev, Danila Sinopalnikov. Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. 2019. Mea- suring compositional generalization: A compre- hensive method on realistic data. ICLR 2020, abs/1912.09713.
Yujian Gan. Xinyun Chen, Qiuping Huang, and Matthew Purver. 2022. Measuring and improving compositional generalization text-to-SQL via com- ponent alignment. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 831- 843, Seattle, United States, Association for Compu- tational Linguistics.
Wuwei Lan, Zhiguo Wang. Anuj Chauhan, Henghui Zhu, Alexander Hanbo Li, Jiang Guo, Shenmin Zhang, Chung- Wei Hang, Joseph Lilien, Yiqun Hu, Lin Pan, Mingwen Dong, J. Wang. Jiarong Jiang, Stephen M. Ash. Vittorio Castelli, Patrick Ng, and Bing Xiang. 2023. Unite: A unified benchmark for text-to-sql evaluation. ArXiv, abs/2305. 16265.
Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL easier to infer from natural language specifications. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2030-2042. Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Dongjun Lee, 2019. Clause-wise and recursive decod- ing for complex and cross-domain text-to-SQL gen- eration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing JEMNLL--JJCNI pages 6045-6051, Hong Kong, China, Association for Com- putational Linguistics.
Chunxi Guo, Zhiliang Tian, Jintao Tang, Pancheng Wang, Zhihua Wen, Kang Yang, and Ting Wang. 2023. case-based reasoning framework for adap- tive prompting in cross-domain text-to-sql.
Jiaqi Guo, Qian Liu, Jian-Guang Lou, Zhenwen Li, Xueqing Liu, Tao Xie, and Ting Liu. 2020. Bench- marking meaning representations neural semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1520-15400 1540, Online. Association for Computational Linguistics.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang. Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng. Nan Huo, Chenhao Ma, Kevin C. Chang, Fei Huang. Reynold Cheng, and Yongbin Li. 2023. Can Ilm already serve as database interface? big bench for large-scale database grounded text- to-sqls, ArXiv, abs/2305.031
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian- Guang Lou, Ting Liu, and Dongmei Zhang, 2019. To- wards complex text-to-SOQ SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4524 4535, Florence, Italy, Association for Computational Linguistics.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross- domain text-to-SQL. semantic parsing. In Findings of the Association for Computational Linguisties: EMNLP 2020. pages 4870-4888, Online, Association for Computational Linguistics.

---

Aiwei Liu,, Xuming Hu, Lijie Wen, and Philip Yu. 2023. A comprehensive evaluation of chatgpt's zero- shot text-to-sql capability.
Griffiths Selinger, M. M. Astrahan, D. D. Chamber- lin, R. A. Lorie, and T. G. Price. 1979. Access path selection in relational database management sys- tem. In Proceedings of the 1979 ACM SIGMOD International Conference on Management of Data, SIGMOD '79, page 23-34, New York, NY, USA. Association for Computing Machinery.
Xiping Liu and Zhao Tan. 2023. Divide and prompt: Chain of thought prompting for text-to-sql.
Sewon Min, Victor Zhong. Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019. Multi-hop reading compre- hension through question decomposition and rescor- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097-6101 Florence, Italy, Association for Compu- tational Linguistics.
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generaliza- tion and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume 1: Long Papers), pages 922-938. Online. Asso- ciation for Computational Linguistics.
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev, 2023. Enhancing few-shot text-to- sql capabilities of large language models: A study on prompt design strategies.
Lysandre Debut Younes Belkada Sayak Paul Sourab Mangrulkar, Sylvain Gugger. 2022. Peft: State-of-the-art parameter-efffcien fine-tuning meth- ods. https: / gi thub. com http://ugginggcce/eeeeeeee
Yilin Niu, Fei Huang. Wei Liu, Jianwei Cui, Bin Wang. and Minlic Huang. 2023. Bridging the Gap between Synthetic and Natural Questions via Sentence De- composition for Semantic Parsing. Transactions of the Association for Computational Linguistics, 11:367-383.
Bailin Wang. Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020a. RAT- SQL: Relation-awant schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578 Online. Association for Computational Linguistics.
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Wei Lee, and Ee-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models.
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text- to-sql with self- colfecoorrction.
Ping Wang, Tian Shi, and Chandan K. Reddy. 2020b. Text-to-sql generation for question answering elec- tronic medical records. In Proceedings of The Web Conference 2020, WWW '20, page 350-361, New York, NY, USA. Association for Computing Machin- ery.
Colin Raffel, Noam Shazcer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text- to-tet transformer. Journal of Machine Learning Research, 21 21(140):1-67.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Lc, and Denny Zhou. 2022. Chain-of of-thuht prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824 24837. Curran Associates. Inc.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau, 2022. Evaluating the text-to-sql capabilities of large language models.
Trina Saparina and Anton Osokin. 2021. SPARQLing Ling database queries from intermediate question decom- positions. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 8984 1999999880088 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: Statc-of-the-at natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Torsten Scholak. Nathan Schucher, and Dzmitry Bah- danau. 2021. PICARD: Parsing incrementally for constrained rugo-segressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901. Association for Computational Linguistics.
Tomer Wolfson, Daniel Deutch, and Jonathan Be- rant. 2022. Weakly supervised text-to-SQL. parsing

---

through question decomposition. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2528-2542, Seattle, United States. Asso- ciation for Computational Linguistics.
Wenting Zhao, Konstantine Arkoudas, Weiqi Sun, and Claire Cardie. 2022. Compositional task-oriented parsing as abstractive question answering. In Pro- ceedings the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4418-4427 Seattle, United States. Association for Computational Linguistics.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoay Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break It Down: A Question Under- standing Benchmark. Transactions the Association for Computational Linguistics, 8:183-199 198.
Victor 7hong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning.
Jingfeng Yang, Haoming Jiang. Qingyu Yin. Danqing Zhang, Bing Yin, and Diyi Yang. 2022. SEQZERO:
Ben Zhou, Kyle Richardson, Xiaodong Yu, and Dan Roth. 2022. Learning to decompose: Hypothetical question decomposition based on comparable texts. ArXiv, abs/2210. 16865.
Few-shot compositional semantic parsing with se- quential prompts and zero-shot models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 49-60, 60, Seattle, United States. Association for Computational Linguistics.
A
Appendix
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language models are versatile decomposers: Decompose evi- dence and questions for table-based reasoning. ArXiv, abs/2301. 13808.
A.1
Q - QPL Model
The Q + QPL model was trained using the Hug- gingFacc Transformers (Wolf et al., 2020) and PEFT (Sourab Mangrulkar, 2022) libraries. As the Flan-T5-XL model was too large to fine-tune on one GPU, we use the LoRA (Hu et al., 2021) method to fine-tune only the and v matrices of the model. Hyperparameters used in training are listed in Table 7. The trained model and training and validation datasets are available on https://hugginggaac.. co/bgunlp
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for ceneral-purposs code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 440-450, Vancouver, Canada. Association for Computational Linguistics.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 201 Sa. Synta Syntax tree networks for complex and cross-domain text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 1653-1663 Brus- sels, Belgium. Association for Computational Lin- guistics.
Hyperparameter
Value
# Epoch
10
Dropout Prob.
0.05
Batch Size
1
Learning Rate
0.0002
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale numan-labeled dataset for complex and cross- cross-domain semantic pars- ing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium. Association for Computational Lirioonaaniinngen Linguistics.
Adaptation (r)
16
LoRA a
32
Table 7: Training Hyperparameters
A.2 Schema Encoding Methods
We compare two methods to describe the database schema when we prompt our models:
Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019. Complex question decomposition for semantic parsing. Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4477-4486, Florence, Italy. Association for Computational Linguistics.
1. Simple Schema Encoding: this is similar to Shaw et al. (2021), i.e. Question Schema Name Table1 Col11, Col12, Table2 Col21, Col22,
Jiajie Zhang, Shulin Cao, Tingjia Zhang, Xin Lv, Jiaxin Shi, Qingwen Tian, Juanzi Li, and Lei Hou. 2023. Reasoning over hierarchical question decomposi- tion tree for explainable question answering. ArXiv, abs/2305.150564
2. Rich Schema Encoding: this encoding pro- vides for each column a simplified type (as in Spider - text, number, date or other); primary and foreign keys; and values.

---

QPL Length
Q
QPL
Q+QD - QPL
Support
87.3%
78.3%
189
2
86.6%
83.4%
277
3
85.3%
78.0%
191
4
75.0%
62.9%
124
67.1%
54.4%
164
48.2%
25.9%
27
31.8%
22.7%
44
28
11.9%
24.4%
18
Overall
77.4%
69.
1034
Table 6: Execution Accuracy of Text-to-QU QPL Models Spider Development Set Length of QPL. QPL length is a more natural measure of query complexity than the method used to classify queries in Spider. We find that there is little correlation between QPL Length and the Spider difficulty level.
For example:
Values are added after each column when an n-gram from the question is found one of the values in one of the rows of the table. For example, for the question "How much does the youngest dog weigh?", the n-gram "dog" is found in the values of the column PetType. In this case, the value annotation PetType text ( dog encoded.
Simple Schema Encoding: pets_
Table Student (StuID, LName, Fname, Age, Sex, Major, Advisor, city_code) Table Pets (PetID, PetType, pet_age, weight)
Table Has_Pet (StuID, PetID)
A.3 Question Decomposer Model
Rich Schema Encoding: pets_
The prompt given to ChatGPT to decompose question given QPI. listed Fig. 5. The full prompt (including BNF and all 6 examples) is available on GitHub.
CREATE TABLE Student (
StuID number, LName text, Fname text, Age number, Sex text, Major number, Advisor number,
A.4 QPL Syntax
We show the BNF of the QPL language we have designed in Fig. 6. This grammar is used part of the PICARD parser used for the decoder of all QPL predictor models.
city_code text, primary key ( StuID )
A.5 Errors by Schema
Table shows error rate for the Q - QPL model with Rich Schema Encoding by schema for cach of the 20 schemas present in Spider's development set.
CREATE TABLE Pets ( PetID number, PetType text ( dog ),
pet_age number, weight number, primary key ( PetID )
We observe that 5 of the 20 schemas (car_1, flight_ 2, student stwdentt_taassenneee trtnscrrattssssss pttsscrrptt_trccknnn. world and wta_ account for 41.8% of the examples in the development and 70.1% of the errors. Most of the errors these specific schemas can be traced to missing key declarations, inconsistent column naming and typing (strings used to encode boolean or number values).
CREATE TABLE Has_Pet (
StuID number, PetID number, foreign key StuID )
references Student ( StuID )
foreign key PetID )
references Pets ( PetID ))

---

QPL is formalism used to describe data retrieval operations over SQL schema in modular manner. A QPL plan is a sequence of instructions for querying tabular data to answer natural language question.
Forget everything you know about SQL, only use the following explanations.
A schema is specified list of <table> specification in the format:
<table>: <comna separated list of columns>
A plan contains sequence of operations.
All operations return stream of tuples.
All operations take input either physical table from the schema (for the Scan operation) the output other operations.
Let's think step by step convert QPL plan natural language plan given schema, question, and QPL that describe the question.
the natural Language plan:
You must have exactly the same number questions as there are steps the QPL. 2. The questions you generate must follow exactly the same order as the steps in QPL.
This the formal specification for each operation:
(BNF goes here, elided for brevity)
Example 1:
Schema:
Table Visitor (ID, Name, Age, Level_of _f__omemmmemssi)m Table Museum ID, Name, Span_Year, Num_of_staff) Table Visit (Visitor_ID, Museum_ID, Total_Spent, Num_of _ickett
Question:
What is the total ticket expense of the visitors whose membership level is 1?
QPL Plan:
#1 = Scan Table [ visitor ] Predicate Le
Output [ ID ]
#2 = Scan Table [ visit ] Output [ visitor_ID Total_spent
#3 = Join #1, #2 Predicate [ #1.10 10 = #2. visitor_ID Output [ #2. Total_sppent ]
#4 Aggregate 43 Output SUM(Total_spent) AS Sum_Total_spent
Natural Language Plan:
#1 = Scan the table Visitor to find who are the visitors with membership level
#2 Scan the table Visit to find what is the total spent by visitors during their visits
Join #1 and #2 to find what is the total spent by each visitor with membership level during their visits
Group Visitor and aggregate the of total spent to find what is the total spent by all visitors with membership level during their visit
examples given)
Now your
Schema:
(schema)
Question: (question)
QPL Plan:
{qpl}
Natural Language Plan:
Figure 5: Prompt used to generate QD

---

<qpl> ::= <line>t
<line> *integer>> <operator>
<operator> ::= <scan>
<aggregate>
<filter>
<sort>
<topsort>
<join>
<except>
<intersect>
<union>
Leaf operator
<scan> ::= Scan Table <table-name> ] <pred>? <distinct>? <output-non-qualif>>
--
Unary operators
<aggregate> Aggregate [ <input> <group-by>? <output--oo--qullif>
<filter> Filter [ <input> <pred> <distinct>? <output-noon-qullif>
<sort>
Sort [ <input> <order-by> <withTie>? <output-non-qualif>
<topsort>
::-
TopSort [ <input> Rows [ <number> ] <order-by>
<withTies>? <output-non-qualif>
Binary operators
<join> ::= Join [ <input>, <input> ] <pred>? <distinct>? soutput-qualifr
<except> :: Except [ <input>, <input> ] <pred> <output-qualif>
<intersect> ::: Intersect [ <input> <input> ] <pred>? <output-qualif?
<union>
Union <input> <input> <output-qualif>
<group-by> GroupBy <column-name> (, <column-name>)*
<order-by> ::= OrderBy [ <column-namen <direction> (, <column-name> <direction>)*
<withTies> ::- WithTies true | false ]
<direction> ::- ASC | DESC
<pred> ::- Predicate [ <comparison> (AND OR <comparison)* *
<distinct> ::: Distinct true false ]
<output--oon-qullif> ::= Output <column-name> <column-name>)+ ]
<output-qualif> ::= Output <qualif--ollmmm--ame>>>>>>>>>>>> (, <qualiff-colum-nnaee>))
<qualif-collummmame> :: <number> <column-name>
Figure 6: QPL Grammar

---

QD-QPL Alignment
Support
Correct
Exec Acc
Avg QPL Gold Len.
[0.0,0.4 0.4
0
5.0
0.51
19
4
21.1
4.6
(0.5, 0.6
2
22.2
6.3
(0.6,
43
9
20.9
5.0
(0.7, 0.8]
63
21
33.3
4.2
(0.8, 0.9]
93
33
35.5
4.6
(0.9, 1]
779
624
80.1
2.8
Table 8: Q+QD - QPI. model trained with QDs predicted by Trained Question Decomposer
Schema ID
Errors
Samples
Error Rate
battle_death
1
16
6%
car_
27
29%
concert_singer
5
45
11%
course_teach
0
30
0%
cre_ cre_Doc_
____Doo__eeme DDoc_Template_aee Mgt
8
84
10%
dog kennels
12
82
15%
employee_hire thir_evaluution
1
38
3%
flight_2 2
15
80
19%
museum visit
2
18
11 %
network_1
9
56
16%
orchestra
0
40
0%
pets_1
2
42
5%
Liker_player
0
40
0%
real gaa__ssaae_properries
1
4
25%
singer
1
30
3%
tracking
16
78
21%
tvshow
5
62
8%
voter_
3
15
20%
world
44
120
37%
wta
15
24%
Grand Total
167
1034
16%
Table 9: Breakdown of errors by Schema ID: 5 schemas of the 20 present in Spider's development set account for 70% of the errors. These schemas do not follow best practices in data modeling and lack proper foreign key declarations.


Reviews:
[{"review_id": "e2evRshrZ3", "reviewer": "Reviewer_fkDx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reaction to \"Reasons to Reject\":**\n\n*R1: Lack experimental results to validate QPL's advantages.*\n\nWe presented a series of experimental results:\n\n1. Similar methods (fine-tuned 3B params LLM with Picard constrained decoding) provide much better results on Hard/Extra-hard queries (+5% overal, +10% for extra-hard specifically).  This is a targeted experiment comparing just the effect of changing the meaning representation from SQL to modular QPL.\n2. In Table 1 - we also compare with a variety of different recent and influential methods on the Spider dataset - NatSQL+RATSQL, Din-SQL GPT4 and a reproduction of recent few-shot GPT3.5 experiments - and observed benefits on complex queries as well.\n3. We also evaluated the capability of experimented programmers to detect whether a predicted query corresponds to a question intent - and showed a significant advantage for QPL over complex SQL.\n4. We demonstrated the capability to train a question decomposer which decomposes a natural language question into simpler steps grounded on the QPL operators.\n\nTaking into account that the paper presents both a method to design the new language (QPL), generate the new formalism\nand the curation of the dataset and training of initial models demonstrating the practical advantage of predicting QPL vs. SQL\nwe believe the contribution for a single paper is significant.  We are in the process of investigating more advantages of QPL - but\nwe still believe that the definition of the formalism, the practical availability of the pipeline producing QPL from SQL and of\nthe Spider-QPL dataset provide value for researchers.\n\n*Reason to reject 2: application scenario for QPL is not well-defined or explored in the paper.*\n\nThis argument is quite general - it is not exactly clear what you mean by \"application scenario\" - assuming we mean \"using QPL as a query representation instead of SQL for researching semantic parsing and issues related to compositional generalization\" - then we believe that the \"Reasons to Accept\" you have listed somehow answer this point. Our motivation is to design an intermediary language that is (1) semantically equivalent to original complex SQL queries, (2) executable, (3) easy to interpret by programmers, (4) easier to predict by ML compositional models. By making available the code for producing Spider-QPL from the existing Spider dataset, we constructively support objectives 1 and 2. The experimental results we report support 3 (with the finetuning Text-to-QPL experiment) and 4 (with the human analysis and the generation of decomposed questions in natural language grounded in QPL). We also believe that the structure of QPL allows more sophisticated training methods than the end-to-end fine-tuning baseline we presented - including methods exploiting question decomposition and auto-regressive training of QPL statements.  \n\nBeyond semantic parsing research, our small user study indicates there is potential for using QPL in a more general \"application scenario\". As you noted, we believe there is more potential in using QPL for more future research around interpretability - we find the fact that programmers cannot detect the validity of SQL statements in over 50% of the cases extremely alarming - and this requires vigorous future research, especially given the rapidity with which code-assistance tools are being deployed in production. \n\n**Reaction to: Reproducibility: 3**\n\nOur github repository includes:\n\n* Code for a fully automatic pipeline to translate the original Spider sqlite data to T-SQL / gather execution plans / translate execution plans to QPL / convert QPL to CTE / validate that the result sets obtained are equivalent to those of the original SQL queries for both training and development sets of Spider.\n* A QPL incremental parser used for the adaptation of the Picard constrained decoding method to QPL and to validate QPL plans interactively.\n* Finetuning code to reproduce the experiments reported in the paper.\n* The Spider-QPL dataset in the form of two JSON files including question, original SQL query, QPL, CTE, result sets for both Dev and Training sets and the Question Decomposition (QD) generated by our question decomposer as well as the QDMR question decomposition produced by the QDMR authors project.\n\nAs such we believe it meets the criteria for a higher reproducibility assessment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The novel QPL offers a systematic approach to decomposing complex SQL queries into manageable sub-queries. The development of a translator leveraging SQL server query optimization plans provides a practical tool for converting SQL queries into QPL, enhancing the applicability of the approach.\n2. Experimental results support the superiority of text-to-QPL parsing over text-to-SQL parsing for equivalent queries, reinforcing the practical significance of the proposed approach.\n3. QPL's ability to be paraphrased as simple questions and its accessibility to non-experts extend its utility beyond semantic parsing, adding interpretability and broader application potential.", "reasons_to_reject": "1. While the paper highlights the advantages of the proposed Query Plan Language (QPL), it appears to lack comprehensive experimental results that robustly validate these advantages. More detailed experiments, including comparisons with existing methods, varied datasets, and a thorough statistical analysis, would provide stronger evidence for the claimed benefits of QPL.\n2. The application scenario for the intermediate representation created by QPL is not well-defined or explored in the paper.  Addressing this aspect would enhance the paper's relevance to practitioners and potential adopters of the approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a modular Query Plan Language (QPL) to decompose SQL queries into regular sub-queries. The authors develop a translator from SQL to QPL and augment the Spider dataset with QPL programs. The experimental results suggest that text-to-QPL parsing is more effective than text-to-SQL parsing for equivalent queries, and QPL offers benefits in question decomposition and interpretability.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "61IfmTtuIo", "reviewer": "Reviewer_wWqA", "summary": "", "questions": "1.  While Q + QD -> QPL is an interesting experiment, why does having additional information hurt as compared to Q -> QPL ? \n\n2. How does In-context learning/few shot learning using large models (as described in Table 1) perform with QPL?\n\n3. Are execution plans the best way of decomposing a SQL query, considering that execution plans are optimized for performance, the paper could analyze whether such optimization aligns perfectly with the interpretability and modularity goals of QPL.?\n\n4. A crucial aspect of QPL is its interpretability. However, it would be valuable to expand on how QPL fares against QDMR concerning interpretability with non-experts. Providing specific examples or user studies to showcase how well non-experts can understand and reason about the queries in each representation could add depth to the paper's discussion on the interpretability aspect of QPL. This comparison could highlight the practical implications and user-friendliness of QPL in real-world scenarios.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answer to questions:**\n\n1. **While Q + QD -> QPL is an interesting experiment, why does having additional information hurt as compared to Q -> QPL?**  \nWe were indeed curious about this point. We believe this aspect is addressed in the paragraph 476-482: The QD we use in the experiment is computed by the Question Decomposer model we used. When using Oracle QD - we do observe much higher performance compared to direct Q->QPL. This experiment motivates further research in learning a more accurate Question Decomposer. See Table 7 in Appendix which indicates that a simple metric of QD/QPL alignment can provide guidance to improve the learning of a better Question Decomposer.\n2. **How does In-context learning/few shot learning using large models (as described in Table 1) perform with QPL?**    \nWe indeed performed the experiment to compare Text-to-QPL using few-shot LLMs as in Table 1 - the results are lower than on SQL using the same prompting strategies. We have performed analysis to explain this discrepancy - working hypotheses at this point are that:\n- (a) GPT3.5 like LLMs \"know\" SQL as part of their pre-training which provides strong prior for specifically generating SQL as opposed to \"unknown languages\" like QPL without further fine-tuning; \n- (b) we suspect the Spider dev dataset is \"memorized\" by these models to some extent (establishing this point is difficult but we tend to believe proving the opposite should be the burden of proof).\n3. **Are execution plans the best way of decomposing a SQL query, considering that execution plans are optimized for performance, the paper could analyze whether such optimization aligns perfectly with the interpretability and modularity goals of QPL.?**   \nThis is a fascinating question - there are indeed different execution plans that can be generated for the same query depending on statistical information gathered by the SQL optimizer. SQL optimizers optimize for performance - and not for \"interpretability\" - although it is complex to model \"interpretability\" in this context. Yet, regardless of the specific plan selected by the optimizer, we do obtain a semantic decomposition of the query into a restricted set of simple operators. In other words, the execution plan achieves \"modularity\" in breaking down complex queries into a plan of simple steps, with a simple uniform semantic of \"stream of tuples flow\" across the nodes of the plan. In further analysis, we have measured the distribution of \"high-level set operations\" which we associate with high-level interpretable structures (union, intersection, set difference) and we identified that these are present in the QPL plans we have computed on the basis of the optimizer data with high frequency. Future research should help us compare different planning strategies when decomposing complex queries to identify those with higher interpretability.\n4. **A crucial aspect of QPL is its interpretability. However, it would be valuable to expand on how QPL fares against QDMR concerning interpretability with non-experts. (...)**   \nQPL is not directly comparable to QDMR because QPL is a semantically equivalent transformation of the original SQL queries, that ensures the exact same results at runtime and is database schema aware. QDMR is more comparable with the QD we obtain from QPL. Unlike QPL, QDMR is not deterministically executable, it requires an additional step of SQL synthesis, where the result is an SQL which is at least as complex as the gold SQL, therefore not more interpretable than the gold SQL.   \nOverall, the comparison of how QPL / QD / QDMR fare in interpretability is definitely an interesting direction for future research - thank you for the suggestion.\n\n**Reaction to \"Reasons to Reject\":**\n\n**R1. The paper compares QPL with SQL - but not QPL with QDMR in terms of interpretability.**\n\nThere is a design decision whether we aim to decompose the natural language question (Q->QD) or the query (SQL->QPL).\nThe two approaches can converge - (SQL->QPL / Q->QDMR). The novelty of our approach in the context of Text-to-SQL is to propose a *semantic* method to decompose complex queries into structured plans. This is in contrast to *syntactic* simplifications of SQL (as in NatSQL). It is also in contrast with *heuristic* question decomposition strategies (as in QDMR) which originate from multi-hop QA.\n\nIn terms of interpretability, as mentioned above, QPL is not directly comparable to QDMR because QPL is an executable programming language. Future research should cover user studies comparing SQL / QPL / QD / QDMR with people at different expertise level in SQL as well as in the database domains (our error analysis indicates people as well as model fail because of lack of domain knowledge).\n\n**R2. Lack of error analysis on predicted QPLs**\n\nWe agree such error analysis will improve the paper - and will be happy to provide it in the paper.\nIt is worth noting that in our experiment we compared SQL to QPL under \"basic conditions\" - that is, without taking into account\nvalues in the database tables and exploiting column types and detailed keys relations. Error analysis indicates that these parameters\nas well as domain specific schema linking knowledge become key systemic contributors to QPL errors.  Our latest experiments confirm proper modeling of these schema parameters plays an important role in QPL prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. By breaking down complex SQL queries into simpler sub-queries, QPL showcases its potential to handle challenging tasks more efficiently. The experimental results provide convincing evidence of QPL's ability to improve the performance of language models (LLMs) when dealing with complex queries.\n\n2. With the use of QPL, the paper argues that the interpretability of the queries is enhanced, allowing for better transparency and understanding of LLMs' outputs.\n\n3. The augmentation of the SPIDER dataset with QPL annotations can be a valuable contribution of this paper. \n\n4. The empirical results presented in the paper provide strong support for the claims made about the usefulness of QPL compared to traditional SQL.", "reasons_to_reject": "1. The paper presents advantages of using the Modular Query Plan Language (QPL) over SQL; however, it lacks a comparison with other decompositions, notably the Query Decomposition Meaning Representation (QDMR), in terms of interpretability. Including such a comparison could provide a more comprehensive understanding of QPL's strengths and weaknesses in relation to existing approaches.\n\n2. The paper would benefit from an error analysis that examines the mistakes made with QPL compared to SQL as a language. This analysis could shed light on specific challenges and issues that arise with QPL, especially considering that it is proposed primarily for the semantic parsing task. Incorporating concrete examples of these mistakes would be helpful for readers to gain deeper insights into the practical implications of using QPL.", "questions_to_authors": "1.  While Q + QD -> QPL is an interesting experiment, why does having additional information hurt as compared to Q -> QPL ? \n\n2. How does In-context learning/few shot learning using large models (as described in Table 1) perform with QPL?\n\n3. Are execution plans the best way of decomposing a SQL query, considering that execution plans are optimized for performance, the paper could analyze whether such optimization aligns perfectly with the interpretability and modularity goals of QPL.?\n\n4. A crucial aspect of QPL is its interpretability. However, it would be valuable to expand on how QPL fares against QDMR concerning interpretability with non-experts. Providing specific examples or user studies to showcase how well non-experts can understand and reason about the queries in each representation could add depth to the paper's discussion on the interpretability aspect of QPL. This comparison could highlight the practical implications and user-friendliness of QPL in real-world scenarios.", "paper_topic_and_main_contributions": "The paper presents the Modular Query Plan Language (QPL), offering an alternative approach to traditional SQL for handling complex queries. By breaking down SQL queries into simpler sub-queries, QPL proposes a modular structure that could potentially benefit Large Language Models (LLMs).\n\nA notable strength of QPL is its emphasis on simplicity and modularity. The authors argue that this design could make it easier for LLMs to learn and comprehend complex logic, which is a crucial aspect in natural language processing. With QPL, LLMs might be able to handle intricate queries more effectively, potentially leading to improved performance.\n\nThe paper claims that LLMs perform better when using QPL compared to regular SQL, supported by evidence from experiments and results. This suggests that QPL might offer advantages over traditional SQL in handling complex queries and improving model accuracy. Furthermore, the paper highlights the potential interpretability benefits of QPL over SQL. The modular design and straightforward syntax of QPL could make it more user-friendly and easier to understand. This aspect is essential in promoting transparency and understanding.\n\nIn addition to proposing QPL, the paper contributes by augmenting the SPIDER dataset with QPL annotations. This augmentation could be a valuable resource for researchers interested in working with modular query languages.\n\nThe proposed text-to-QPL model, which utilizes a fine-tuned FlanT5-XL with a constrained decoding algorithm, demonstrates the practical applicability of QPL.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The related works are adequately cited.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper reads well.\n\nMinor\n\n1. line 462 - use consistent notation for the model (FlanT5-XL)."}, "extra_text": {}}, {"review_id": "Fly5KNvo1o", "reviewer": "Reviewer_zf7K", "summary": "", "questions": "A. While a query optimizer is used to convert SQL queries into execution plans, it is not mentioned how these plans are converted to the proposed QPL language. Other than that, it is also unclear how QPL is translated to CTE and make them executable?\n\nB. How do you validate QPLs have the same execution results as the original SQL queries? Is it a logical proof or empirical test? If the latter, what is the data you use to validate, and what is the coverage of QPL?\n\nC. Why are there only 6,509 training examples? Spider has 7,000 training examples.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answers to Questions:**\n\n**A. While a query optimizer is used to convert SQL queries into execution plans, it is not mentioned how these plans are converted to the proposed QPL language. Other than that, it is also unclear how QPL is translated to CTE and make them executable?**\n\nThe process of translation \"execution plan\" -> QPL and QPL -> CTE are standard \"programming language\" transformations. The key design decision in this process are those listed in the article - which consist of abstracting away physical and index parameters from the execution plan and making sure QPL is \"context free\" for those operators (such as \"nested loops\") which are not context-free in the SQL Server execution plan language. As for executing QPL, the key decision is to rely on the extremely useful CTE feature of SQL.\n\n* The translation from SQL Server execution plans (obtained as XML documents) into QPL is described in lines 364-371. The process of translation is implemented as a Scala program - with a parser of the XML plans (200 lines of type definitions and 400 lines of parsing code) and an AST to AST recursive translator (800 lines of Scala code covering all types of operators in the SQL Server execution plan syntax).  \n* The translation from QPL to CTE is implemented in Python (400 lines) - it produces a sequence of simple SQL queries that are bound together into a single CTE expression.\n\nOur github provides this code as well as notebooks and interactive Web User Interface tools to experiment with these transformations on arbitrary queries.\n\n**B. How do you validate QPLs have the same execution results as the original SQL queries? Is it a logical proof or empirical test? If the latter, what is the data you use to validate, and what is the coverage of QPL?**\n\nThe pipeline consists of translating original SQL queries to execution plans (we rely on the correctness of the SQL optimizer at this stage), then translate each step in the execution plans into QPL queries which are simplified syntactic realizations of simple SQL constructs (one for each QPL primitive operator), which are then bound into a CTE expression.\n\nWe validated empirically that this transformation preserved execution results on the Spider dataset (as well as on augmented tests which are not reported in the paper) - one the 6,509 training samples that we successfully converted from SQLite to T-SQL (see question C) and the 1,034 samples from the dev set.  In some cases, the Spider databases contain little data and some queries return empty result sets.  In these cases, we augmented the content of the databases so that all queries return non-empty result sets and we confirmed that the result sets are equivalent.\n\nC. Why are there only 6,509 training examples? Spider has 7,000 training examples.\n\nThe results we report are based on an automatic translation pipeline from the original Spider dataset in SQLite to QPL.\nThis translation pipeline failed on the missing 491 samples for a variety of reasons:\n\n* The original SQLite query could not be translated into a valid SQL Server query because of the \"non strict\" nature of SQLite (no type checking, no enforcement of aggregation rules, loose implementation of referential integrity)\n* The generated execution plan included a rare configuration that we did not cover as part of our transformation logic.\n\nWe found that the identification of these problematic SQLite cases through translation to a strict T-SQL variant is a valuable contribution to better analyze Spider results - and we will add information about this process in Appendix.\n\n**Reaction to Reason to Reject:**\n\n**R1. The experiments setup cannot sufficiently show that QPL is an effective MR.**\n**(1) The authors uses FlanT5 for their methods while using T5 for the baseline T5+PICARD method. It is hard to tell whether the performance gain is brought by QPL or instruction-tuning in FlanT5 itself.**\n\nWe actually did use the same FlanT5-XL model for finetuning the Q->SQL with Picard as we used for QPL - thank you for noting this discrepancy.  We will clarify it in the paper.  This experiment is intended to compare SQL with QPL as a target language \"all other things being equal\".\n\n**(2) The authors do not compare the proposed method with existing work in the main experiment (Table 3), \nsuch as QPL vs NatSQL and QPL -> QD vs QDMR. It is unclear if the proposed methods are better or as effective as existing ones.**\n\nWe do report baselines for QPL vs. NatSQL in Table 1 - as well as other recent LLM-based few-shot methods (Din-SQL GPT4 as well as GPT-3.5) under similar experimental settings as our experiments (Spider Dev Set split by difficulty level).  The experiment we report in Table 3 is a targeted experiment where we specifically compare SQL vs. QPL as a target language \"all other things being equal\" - using the same constrained decoding method (Picard adopted to QPL with an incremental QPL parser we developed).\n\nQDMR is not a query language - it is used as a set in a question decomposition approach. We did compare (Q+QDMR->QPL) vs (Q+QD=>QPL) and obtained lower results than with (Q+QD) - we did not report this experiment as it did not bring direct clarity on the contribution, compared to the discussion of Oracle QD we report in lines 476-483.\n\n**(3) The experiments are all on the Spider dataset. Since MRs should be generally applicable to the text-to-SQL task, \nit would be very helpful if the authors can include comparison with open-sourced methods on another text-to-SQL dataset, \nsuch as KaggleDBQA (Lee et al., 2021), EHRSQL (Lee et al., 2023), and BIRD (Li et al., 2023).**\n\nIn principle, it is indeed better to test on more databases - but it is not clear how spending resources on testing on more datasets would change the overall contribution of this work. KaggleDBQA is a nice dataset, but it has had relatively little impact on the community (few citations compared to Spider and few comparisons with it across different methods); it also focuses on demonstrating the contribution of database documentation to few-shot LLM-based semantic parsers - which is interesting but orthogonal to the aspect of query complexity we discuss in this contribution; EHRSQL is a single schema dataset (compared to ~150+ different schemas in different domains in Spider dev+training sets); BIRD is a very recent contribution (the paper is dated May 2023) and focuses on using values in the databases (which we decided not to do in our first experiments to compare with previous work on Spider).\n\n**R2. In terms of the user study, the authors draw their conclusion based on p < 0.06, while in such tests p < 0.05 is generally considered statistically significant.**\n\nIndeed 0.05 is the standard - yet, 0.06 is not a baseless observation either (see <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4111019/> for a discussion of p-value fallacies).\n\n**R3. This paper lacks analysis on the experiment results. It would be very helpful if the author can provide more insight into why QPL is \neasier to learn and interpret, especially when compared with existing methods.**\n\nWe discuss this point in our motivation section (lines 273-294) in our comparison with existing intermediary representation methods (mainly NatSQL) - learning a more compositional and regular representation of queries allows models to generalize better to more complex queries. More experiments are necessary to \"explain\" why a modular query language is easier to interpret and learn than a non-modular one - but contributing the QPL language, the pipeline to produce an experiment on ~7,500 (query, question) pairs and presenting supporting evidence that modularity helps in learning and interpretation is a first step in obtaining such explanations in the future.\n\n**Presentation Improvements:**\n\n> The first two sections (introduction and related work) use more than half of the main text (4.5 pages)\n\nThese two sections include critical material related to our approach and are not just survey of previous work:\n\n* Figure 1 illustrates our approach (2/3 of page 2)\n* Sections 2.3 and 2.4 compare existing approaches to intermediate representations to our approach and to QD.  These contribute to the presentation of our approach and are not only a survey of previous work.\n\n**Reproducibility: 2**\n\nOur github repository includes:\n\n* Code for a fully automatic pipeline to translate the original Spider sqlite data to T-SQL / gather execution plans / translate execution plans to QPL / convert QPL to CTE / validate that the result sets obtained are equivalent to those of the original SQL queries for both training and development sets of Spider.\n* A QPL incremental parser used for the adaptation of the Picard constrained decoding method to QPL and to validate QPL plans interactively.\n* Finetuning code to reproduce the experiments reported in the paper.\n* The Spider-QPL dataset in the form of two JSON files including question, original SQL query, QPL, CTE, result sets for both Dev and Training sets and the Question Decomposition (QD) generated by our question decomposer as well as the QDMR question decomposition produced by the QDMR authors project.\n\nAs such we believe our contribution meets the criteria for a higher reproducibility assessment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper attempts to address a salient research question in semantic parsing: meaning representation (MR). The proposed QPL is intended to serve as an MR that reduces model learning difficulty and help non-expert users to identify mistakes in complex SQL queries. The idea of decomposing natural language questions using QPL is interesting, although not sufficiently justified.", "reasons_to_reject": "I have three major concerns for this paper:\n1. The experiments setup cannot sufficiently show that QPL is an effective MR. (1) The authors uses FlanT5 for their methods while using T5 for the baseline T5+PICARD method. It is hard to tell whether the performance gain is brought by QPL or instruction-tuning in FlanT5 itself. (2) The authors does not compare the proposed method with existing work in the main experiment (Table 3), such as  QPL vs NatSQL and QPL -> QD vs QDMR. It is unclear if the proposed methods are better or as effective as existing ones. (3) The experiments are all on the Spider dataset. Since MRs should be generally applicable to the text-to-SQL task, it would be very helpful if the authors can include comparison with open-sourced methods on another text-to-SQL dataset, such as KaggleDBQA (Lee et al., 2021), EHRSQL (Lee et al., 2023), and BIRD (Li et al., 2023).\n2. In terms of the user study, the authors draw their conclusion based on p < 0.06, while in such tests p < 0.05 is generally considered statistically significant.\n3. This paper lacks analysis on the experiment results. It would be very helpful if the author can provide more insight into why QPL is easier to learn and interpret, especially when compared with existing methods.\n\nThis paper lacks elaboration on many important details. Please see the questions listed below.", "questions_to_authors": "A. While a query optimizer is used to convert SQL queries into execution plans, it is not mentioned how these plans are converted to the proposed QPL language. Other than that, it is also unclear how QPL is translated to CTE and make them executable?\n\nB. How do you validate QPLs have the same execution results as the original SQL queries? Is it a logical proof or empirical test? If the latter, what is the data you use to validate, and what is the coverage of QPL?\n\nC. Why are there only 6,509 training examples? Spider has 7,000 training examples.", "paper_topic_and_main_contributions": "This paper proposes Query Plan Language (QPL), a new meaning representation for text-to-SQL parsing that can be adapted for decomposing natural language questions (QD). The paper claims that QPL is easier for models to learn and for non-expert users to understand.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "While the paper presents a comprehensive review of related work, he first two sections (introduction and related work) use more than half of the main text (4.5 pages), leaving limited space for the authors to introduce their methodology and discuss their experiments in detail. This lack of details hinders the soundness and reproducibility of the paper."}}]

Author rebuttal:

Meta-review: This paper presents a query plan language to decompose sql queries. The problem is text-to-sql translation / parsing. The method is sound. I find the decomposition strategy to be sound, and it could potentially bring some interpretability down the road. The results demonstrate the model is effective.

The reviewers pointed out a few suggestions to strengthen the paper: running experiments with more (and commonly used) datasets, and more importantly, analysis describing what kind of queries their method is most effective with (which they shared in the discussion, so it would be easy to add it to the camera-ready version).

Not running experiments with papers published in 2023 is not negative, as they are considered contemporaneous. This is minor and easily fixable, but claiming p-value < 0.06 goes against the norm. The results are just not statistically significant (and that is probably fine). Finally, the authors are committing to release their implementation, so it is hard to justify anything but a high score for Reproducibility.
Decision: Accept-Findings