Error Detection for Text-to-SQL Semantic Parsing
Authors: Shijie Chen
Ziru Chen
Huan Sun
Yu Su
Venue: EMNLP 2023 Findings (2023)
Keywords: Semantic Parsing
Text-to-SQL
Error Detection

Abstract: Despite remarkable progress in text-to-SQL semantic parsing in recent years, the performance of existing parsers is still far from perfect. 
Specifically, modern text-to-SQL parsers based on deep learning are often over-confident, thus casting doubt on their trustworthiness when deployed for real use. In this paper, we propose a parser-independent error detection model for text-to-SQL semantic parsing.
Using a language model of code as its bedrock, we enhance our error detection model with graph neural networks that learn structural features of both natural language questions and SQL queries. We train our model on realistic parsing errors collected from a cross-domain setting, which leads to stronger generalization ability. Experiments with three strong text-to-SQL parsers featuring different decoding mechanisms show that our approach outperforms parser-dependent uncertainty metrics. Our model could also effectively improve the performance and usability of text-to-SQL semantic parsers regardless of their architectures.

Error Detection for Text-to-SQL Semantie Parsing
Shijie Chen
Ziru Chen Huan Sun Yu Su
The Ohio State University
{chen. 18216, chen. 8336, sun. 397, 809)eosu. edu
Abstract
even for experts (Jorgensen and Shepperd, 2007; Weiss et al., 2007). Therefore, an accurate error de- tector that can flag parsing issues and accordingly trigger error correction procedures (Chen et al., 2023) can contribute to building better natural lan- guage interfaces to databases.
Despite remarkable progress in text-to-SQL se- mantic parsing recent years, the performance of existing parsers still far from perfect. Specifically, modern text-to-SQL. parsers based on deep learning are often over-confident, thus casting doubt their trustworthiness when de- ployed for real use. In this paper. we propose a parser-independent еrror detection model for texi-to-SQI. semantic parsing. Using a lan- guage model of code as bedrock, we enhance emor detection model with graph neural net- works that learn structural features of both nat- ural language questions and SQL queries. We train model realistic parsing errors col- lected from a cross-domain setting, which leads stronger generalization ability. Experiments with three strong text-to-SQI. parsers featur- ing different decoding mechanisms show that our approach outperforms purser-dependent un- certainty metrics. Our model could also effec- tively improve the performance and usability of text-to-SQI. semantic parsers regardless of their architectures'
Researchers have proposed multiple approaches for error detection in text-to-SQL parsing, Yao et al. (2019, 2020) detect errors by setting threshold on the prediction probability or dropout-based un- certainty of the base parser. However, using these parser-dependent metrics requires the hase parser to be calibrated, which limits their applicability. Sev- eral interactive text-to-SQL. systems detect parsing errors based on uncertain span detection (Gur et al.. 2018; 8; Li et al., 2020; Zeng et al., 2020). Despite having high coverage for errors, this approach is re- ported to be of low precision. Finally, text-to-SQL re-rankers (Yin and Neubig, 2019; Kelkar et al., 2020; Bogin al., 2019; Arcadinho al., 2022), which estimate the plausibility of SQL predictions. can seen as on-the-fly error detectors. Neverthe- less, existing re-rankers are trained on in-domain parsing errors, limiting their generalization ability.
1 Introduction
Recent years have witnessed renewed interest in text-to-SQL semantic parsing (Bogin et al., 2019; Lin et al., 2020; Wang et al., 2020; Rubin and Berant, 2021; Cao et al., 2021; Gan et al., 2021; Scholak et al., 2021; Qi et al., 2022; Li al., 2023), which allows users with a limited technical back- ground to access databases through natural lan- guage interface. Although ttate-of-the-art semantic parsers have achieved remarkable performance on Spider YYu et al., 2018), large-scale cross-domain text-to-SQL benchmark, their performance still far from satisfactory for real use. While syntax er- rors can be automatically caught by SQL execution engines, detecting semantic errors in executable SQL queries can be non-trivial and time- come-consuming
In this work, we propose a generalizable and parser-independent error detection model for text- to-SQL semantic parsing. Since syntax errors can be easily detected by execution engine, fo- cus on detecting semantic errors in executable SQL predictions. We start developing our model with CodeBERT (Feng et al., 2020), a language model pre-trained on multiple programming languages. On top of that, we use graph neural networks to cap- ture compositional structures in natural language questions and SQL queries to improve the perfor- mance and generalizability of our model. We train our model parsing mistakes collected from a realistic cross-domain setting, which is indispens- able to the model's strong generalization ability. Furthermore, we show that our model versatile and can be used for multiple tasks, including er-
'Our implementation is available at tttps: Tem/OSUU-NL--GruupTeee Becess--Errr-oee peror-Detecctin

---

ror detection, re-ranking, and interaction triggering, To summarize, our contributions include:
2.2 Re-ranking for Text-to-SQL Parsing
Noticing the sizable gap between the accuracy and beam hit rate of semantic parsers, researchers have explored building re-ranking models to bridge this gap and improve parser performance. Global- GNN (Bogin et al., 2019) re-ranks beam predic- tions based on the database constants that appear in the predicted SQL query. This re-ranker is trained together with its base parser. More recently, Bertrand-DR (Kelkar al., 2020) and T5QL (Ar- cadinho al., 2022) fine-tune pre-trained lan- guage model for re-ranking. However, both report directly re-ranking all beams using re-ranker scores hurts performance. To get performance gain from re-ranking, Bertrand-DR only raises the rank of prediction if its re-ranking score is higher than the preceding one by threshold. T5QL com- bines re-ranking score and prediction score by weighted sum. Both approaches require tuning hyper-panee prpee-parameters. In contrast, when directly using the proposed parser-independent error detection model as re-rankers, we observe performance im- provement on BRIDGE v2 and NatQQL without any constraint, showing that our approach is more generalizable and robust.
We propose the first generalizable and parser- independent error detection model for text-to- SQL parsing that is effective on multiple tasks and different parser designs without any task- specific adaptation. Our evaluations show that the proposed error detection model outper- forms derser-dependent uncertainty metrics and could maintain its high performance un- der aross-parser evaluation settings.
Our work is the first comprehensive study on error detection for text-to-SQL. parsing. We evaluate the performance of error detection methods on both correct and incorrect SQL predictions. In addition, we show through simulated interactions that a more accurate error detector could significantly improve the efficiency and usefulness of interactive text- to-SQL parsing systems.
2 Related Work
2.1 Text-to-SQL Semantic Parsing
Most existing neural text-to-SQL parsers adopt three different decoding mechanisms. The first with constrained de-
2.3 Interactive Text-to-SQL Parsing Systems
one is coding, where parser models query synthesis as sequence generation task and prunes syntacti- cally invalid parses during beam search. Several strong text-to-SQL parsers apply this simple idea, including BRIDGE v2 (Lin et al., 2020), PICARD (Scholak et al., 2021), and RESDSQL (Li et al., 2023). Another popular decoding mechanism is grammar-based decoding (Yin and Neubig, 2017). where parsers first synthesize an abstract syntax tree based on a pre-defined grammar and then con- vert it into a SQL query. Parsers using intermediate representations, such IR-Net (Guo et al., 2019) and NatSQL (Gan et al., 2021) also fall into this category. Grammar- based decoding ensures syntac- tic correctness but makes the task harder to learn due to the introduction of non-terminal syntax tree nodes. Different from the above autoregressive de- coding strategies, SmBoP (Rubin and Berant, 2021) applies bottom-up decoding where SQL query is synthesized by combining parse trees of different depths using a ranking module. We evaluate our model with semantic parsers using each of these three decoding strategies and show that our model is effective on all of them.
Interactive text-to-SQL parsing systems improve the usability of text-to-SQL semantic parsers by correcting potential errors in the initial SQL predic- tion through interactive user feedback. MISP (Yao et al., 2019, 2020) initiates interactions by setting a confidence threshold for the base parser's predic- tion probability. While this approach is intuitive, it requires the base parser to be well-calibrated when decoding, which does not hold for most mod- ern parsers using deep neural networks. In addi- tion, this design hardly accommodate some recent parsers, such as SmBoP (Rubin and Berant, 2021), whose bottom-up decoding mechanism does not model the distribution over the output space. Several other interactive frameworks (Gur et al., 2018; Li et al., 2020; Zeng et al., 2020) trigger interactions when an incorrect or uncertain span is detected in the input question or predicted SQL query. While these approaches have high coverage for parsing errors, they tend trigger unnecessary interactions for correct initial predictions. For ex- ample, PITA (Li et al., 2020) triggers interactions on 98% of the questions on Spider's 'ss development set when its base parser has an accuracy of 49%.

---

Train
Development
Test
Parser
#Beam
Hit
Miss
#Beam
Hit
Miss
#Beam
Hit
Miss
SmBoP
5322
6062/1 4
12937/2.6
1416
1864/1.3 33
3159/2.5
989
1324/1.3
1498/1.5
BRIDGE v2
5482
3898/0.7
8932/1.6
1465
1116/0.8
2057/1.4
1031
852/0.8
1096/1.1
NatSQL
5398
7095/1.3
13443/2.5
1474
2207/1.5
3522/2.4
1030
1582/1.5
2584/2.5
Table 1: Statistics of the data collected from three base parsers. #Beam: number of beams that have executable predictions. For beam hit and misses, we report total/average__ett bear
Compared to these methods, the proposed method strikes better balance between performance and efficiency, and thus could improve the user experi- ence of interactive text-to-SQL parsing systems.
top predictions are skipped. We report the num- ber of beams, total number of question-SQL pairs, and average number of such pairs per beam for each split in Table 1. Following existing litera- ture (Kelkar et al., 2020), we refer to correct SQL queries in the beam as beam hits, and incorrect ones as beam misses.
3 Parser-independent Error Detection
3.1 Problem Formulation
We notice that BRIDGE v2 generates signifi- cantly fewer executable SQL queries all data splits. This is due its unconstrained decoder with rule-based filtering. In addition that, BRIDGE v2 generates a default SQL query that counts the number of entries the first table when there are no valid predictions in the beam. Although this mechanism ensures the parser generates at least one executable query, such negative samples do not fit the overall error distribution and may harm the error detection model. NatSQL and SmBoP take into account grammatical constraints of SQL during decoding and thus could generate more ex- ecutable queries than BRIDGE v2. Table also shows that NatSQL and SmBoP produce a similar amount of beam hits and beam misses on the train- ing and development splits. However, the number of executable beam misses generated by SmBoP on the test split is noticeably lower, while the behavior of NatSQL is more consistent.
Given question X = (21.,2,"..,.0 and SQL query أ = {ύ1, i2... ,m. predicted by text-to-SQL parser, the error detection model esti- mates the probability of y being correct, denoted by
pςğ = y0 xx
We perform error detection and action triggering by setting threshold for s. For re-ranking, we directly use as the ranking score without modification.
3.2 Cross-domain Error Collection
We consider two factors that could lead to text-to- SQL parsing errors: insufficient training data and the cross-domain generalization gap. To simulate such errors, we collect data from weak versions of base parsers in a cross-domain setting. More specifically, we split the Spider training set into two cqual-sized subsets by databases and train the base parser on each subset. Then we perform in- ference on the complementary subset and collect beam predictions data for error detection. We keep executable SQL queries and label them based on execution accuracy. We use fixed version Spider's official evaluation script (Appendix A) and keep up to five parser predictions for each ques- tion after deduplication. The collected samples are divided into training and development sets by am 80:20 ratio according to databases as well. In this way, get high-quality training data for our error detection model in setting that approximates the real cross-domain testing environment. For testing, we train each base parser on the full Spider training set and collect executable beam predictions on the Spider development set. Beams with un executable
3.3 Model Architecture
Figure 1 illustrates the architecture of the pro- posed error detection models. We use Code- BERT (Feng et al., 2020) as our base encoder to jointly encode the input question and SQL query. Following CodeBERT's input construc- tion during pre-training, we concatenate questions and SQL queries with special tokens, namely [CL.S), Im, [SFP), 01, i....., Dn, [FOS) as input and obtain their contextualized representa- tions hx and hg. We only use question and SQL as input since we found in preliminary experiments that adding database schema information (table and column names) in the input hurts performance.
In light of the compositional nature of questions

---

Finally, 2-layer feed-forward neural network with tanh activation is used score the aggre- gated representation v. The score s for each input question-SQQ -SQit pair is:
GAT
= p( = x, y oFFN (holobal)
Apps
hru.
Pipcon
Agi
where y" is the gold SQL query and represents the sigmoid function. We train our model by mini- mizing binary cross entropy loss:
(CLS] How
have (SEP) SELECT court
FROM singer (EC6)
Question
= 1g==y log + lg==-) log(1 -
Figure I: Architecture of our emor detection models.
During training, we supply the model with samples from K beams each step, where K is the batch size.
and SQL queries, we propose model their struc- tural features via graph neural networks. For nat- ural language questions, we obtaín their depen- dency parse trees and constituency parse trees from Stanza (Qi et al., 2020) and merge them together. This is possible since edges in dependency parse trees are between two actual tokens, which corre- sponds to leaf nodes in constituency parse trees. For SQL queries, wC extract their abstract syn- tax trees via Antlr4.2 To make the input graphs more compact and lower the risk of overfitting, we further simplify the parse trees by removing non-terminal nodes that only have one child a top-down order. Additionally, for SQL queries, we remove the subtrees for join constraints which do not carry much semantic information but are often quite long. At last, add sequential edges con- necting the leaf nodes in the parse trees by their order in the original questions and SQL queries preserve natural ordering features during graph learning.
Experiments
In this section, we first evaluate the performance (Section 4.2.1) and generalization ability (Section 4.2.2) of our error detection model on the binary detection task. Then we investigate effectiveness when used for re-ranking (Section 4.2.3) and action triggering (Section 4.2.4).
model's
4.1 Experiment Setup
Baseline Methods We compare our parser- independent error detectors with parser-dependent uncertainty metrics, including prediction probabil- ity and dropout- based uncertainty. Since SmBoP (Rubin and Berant, 2021) uses bottom-up decoding which separately scores and ranks each candidate prediction, we deduplicate SmBoP's beam predic- tions by keeping the maximum score and perform softmax on the deduplicated beam to get a probabil- ity distribution over candidate predictions, which can be seen as a reasonable approximation to its confidence. BRIDGE v2 (Lin et al., 2020) and Nat- SQL (Gan et al., 2021) use autoregressive decoders, and we directly use the log probability of its pre- diction its confidence score. Probability- based methods are denoted by superscript p. terms of dropout- based uncertainty, we follow MISP (Yao et al., 2019) and measure the standard deviation of the scores (SmBoP) or log probability (BRIDGE v2 and NatSQL) of the top-ranked prediction in 10 passes. Dropout Dropout-based uncertainty is denoted by superscript s.
We initialize the representations parse tree leaf nodes with CodeBERT's contextualized repre- sentations and randomly initialize representations of other nodes according to their types in the parse tree. The two input graphs encoded by two sep- arate 3-layer graph attention networks (Brody et al.. 2022). Then we obtain the global representation of each graph via average pooling and concatenate them to get an aggregated global representation:
gloobll
globeal
global,
We denote models with graph encoders as Code- BERT+GAT'n in Section 4. When simply fine-tuning CodeBERT, globbll hCCLs)-
Evaluation Metries on the error detection task. After that, we test performance when is used for re-ranking and action triggering.
We first evaluate our model
AAntlr: https://www.ans cottrr.org/, more details in Ap-
pendix D.

---

Positive
Negative
Parser
Model
Acc
AUC
Precision
Recall
F1
Precision
Recall
F1
SmBoPP SmBoP"
80.8
94.5
86.6
52.1
18.9
23.9
77.4
67.0
81.5
91.9
85.7
56.6
25.3
29.4
76.9
79.2
SmBoP
CodeBERT
82.9
92.6
86.7
60.8
33.9
36.3
78.3
80.8
CodeBERT+GAT
85.0
90.6
87.2
56.7
44.4
46.4
79.8
81.7
BRIDGE v2" BRIDGE v2*
77.9
85.1
80.9
60.9
50.4
54.0
73.2
77.8
76.1
79.5
57.9
46.0
50.8
71.3
77.7
BRIDGE v2
CodeBERT CodeBERT+GAT
78.6
87.6
82.4
69.1
51.1
$7.0
75.3
81.3
80.9
86.4
83.4
69.5
58.7
63.2
77.6
83.0
N NaiSQL
78.1
93.2
84,6
67.3
36.1
45.4
76.3
79.2
77.0
91.4
83.0
62.8
33.1
40.3
74.0
76.2
NatSQL
CodeBERT
84.6
90.8
87.3
72.3
60.5
64.6
81.8
86.5
CodeBERT+GAT
86.6
87.4
86.8
68.5
68.1
67.0
81.7
86.9
Table 2: Error detection performance with three base parsers on Spider's development set. We highlight the best performance with each parser in bold.
For error detection, we report precision, recall, and Fl scores for each method on both positive and negative samples. However, these metrics depend on the threshold used. To more comprehensively evaluate the overall discriminative ability of each method, we present the area under the receiver operating characteristic curve (AUC), which is not affected by the choice of threshold. We apply 5- fold cross- validatil and report performance using the threshold that maximizes the accuracy of each method. Test samples partitioned by databases. For the re-ranking task, we evaluate on the final beam predictions of fully trained base parsers on Spider's 's development set and report top-1 accuracy.
4.2 Results
4.2.1 Error Detection
To evaluate error detection methods realistic setting, we use final SQL predictio made by Sm- BoP, BRIDGE v2, and NatSQL on Spider's devel- opment set that are executable as test datasets. As shown in Table 2, the dropout-based uncertainty measure significantly outperforms the approximate confidence measure negative samples (29.4 23.9 in negative FI) with SmBoP. However, we notice the opposite with BRIDGE v2 and NatSQL, which consistent with the observation of MISP (Yao et al., 2019) that is based on parser with autoregressive decoder as well. Nonetheless, the dropout-based uncertainty measure is still indica- tive of these two parsers. We also notice that parser- dependent metrics exhibit high recall and low pre- cision on positive samples, showing that the three parsers, despite using different decoding strategies, are over-confident in their predictions.
For action triggering, we evaluate system perfor- mance under two settings: answer triggering and interaction triggering. answer triggering, we measure system answer precision when answering different numbers of questions. In interaction trig- gering, we measure system accuracy using different numbers of interactions.
Error detection and re-ranking results are aver- age performance over different random seeds. For action triggering, we evaluate checkpoints with the highest accuracy on the development split of our collected data.
On all three parsers, our proposed error detector significantly outperforms the two parser-dependent uncertainty metrics, especially on negative sam- ples. With the added structural features, Code- BERT+GAT further improves overall error detec- tion performance, especially in recall on incorrect predictions (8.6% absolute improvement on av- erage), which indicates structural features could help the model learn more generalizable error pat- terns. We also find that the advantage of Code- BERT+GAT mainly comes from higher perfor- mance on hard and extra hard questions (Appendix C). Compared to depeededt metrics, our
Implementation Our models are trained with batch size of 16 and optimized by the AdamW (Loshchilov and Hutter, 2019) optimizer with de- fault parameters. Training lasts 20 epochs with a learning rate of 3e-5 following a linear decay schedule with 10% warm-up steps. All models are trained on an NVIDIA RTX A6000 GPU.

---

Positive
Negative
Target
Source
Model
Acc
AUC
Precision
Recall
F1
Precision
Recall
F1
SmBoPP SmBoP"
80.8
94.5
86.6
52.1
18.9
23.9
77.4
67.0
81.5
91.9
85.7
56.6
25.3
29.4
76.9
79.2
CodeBERT
80.3
94.8
86.4
52.9
16.3
20.3
77.2
75.9
SmBoP
BRIDGE v2
CodeBERT+GAT
80.0
95.9
86.8
44.1
14.4
18.8
77.8
77.0
CodeBERT CodeBERT+GAT
81.9
87.7
58.2
23.4
30.8
79.6
75.9
NatQQL
83.1
93.3
87.6
56.0
31.5
38.1
79.9
78.2
BRIDGE v2 BRIDGE v2"
77.9
85.1
80.9
60.9
50.4
54.0
73.2
77.8
76.1
83.6
79.5
57.9
46.0
50.8
71.3
77.7
CodeBERT CodeBERT+GAT
78.4
88.2
82.4
70.1
50.1
55.6
75.0
82.4
BRIDGE v2
SmBoP
80.2
87.9
83.5
71.5
55.5
61.1
77.1
83.1
CodeBERT CodeBERT+GAN
78.5
88.2
82.9
68.6
50.1
56.9
75.9
80.0
NatSQI
78.9
87.2
82.5
68.2
51.7
56.9
75.5
81.7
Na P NaiSQL
78.1
93.2
84.6
67.3
36.1
45.4
76.3
79.2
77.0
83.0
62.8
33.1
40.3
74.0
76.2
CodeBERT CodeBERT+GAT
83.7
86.1
84.0
63.5
61.1
58.5
77.0
85.2
NatSQL
SmBoP
83.5
87.2
84.7
65.1
59.5
59.9
78.2
85.7
CodeBERT
79.7
89.7
84.2
65.3
43.3
51.1
76.9
79.6
BRIDGE v2
CodeBERT+GAT
81.1
89.2
84.8
65.8
48.0
54.7
78.1
81.5
Table 3:
Cross-parse generalization performance with three base parsers Spider's development set. We highlight the best performance with each target parser bold.
model yields the largest performance gain in both accuracy and AUC with NatSQL and reasonable gains with the other two parsers, possibly due to the higher quality of its training data and better behavior consistency on the test split.
v2 and NatSQL, two autoregressive parsers, more likely share prefixes and differ in simple operations, such as the choice of columns, aggre- gation functions, or logic operators (examples Appendix E). In contrast, the bottom-up decoder of SmBoP generates more diverse errors with com- plex structures, such as subqueries and set opera- tions. The higher diversity of SmBoP's predictions increases the coverage of the data collected from it, which contributes the stronger generalization ability of the corresponding error detectors.
4.2.2 Cross-parser Generalization
We evaluate our models' cross-parser generaliza- tion ability by training етrоr detectors on data col- lected from one parser and testing on the other two following the same 5-fold cross-validation setting. Table summarizes cross-parser transfer perfor- mance on each parser. Even in this setting, our error detectors could still outperform parser-dependent metries except for SmBoP, where our models fall short slightly in AUC.
4.2.3 Re-ranking
Re-ranker
SmBoP
BRIDGE v2
NatSQL
N/A
75.1
67.3
71.3
RR
72.2
On all parsers, we observe better performance on models trained with stronger parsers. For ex- ample, on SmBoP, the CodeBERT+GAT model trained with NatSQL is better than the one trained with BRIDGE v2 by 2.1 2.1% in accuracy and 1.2% in AUC. Meanwhile, the models trained with SmBoP perform the best on BRIDGE v2 and NatSQL in negative F1, accuracy, and AUC. We hypothesize errors made by stronger parsers are more diverse and of higher quality and thus allow models trained on them to generalize better to weaker parsers. We found that the errors generated by BRIDGE
ED + RR
73.7
68.9
73.9
Beam Hit
80.5
73.1
81.1
Table 4: Execution accuracy with re-ranking using the CodeBERT GAT model. RR: Re-ranking all beams; ED+RR: Re-ranking beams after error detection.
We evaluate the re-ranking performance of our error detection models in two settings. In ranking-all (RR), we re-rank all beams based on the score assigned by the error detector. In error de- tection then re-ranking (ED+RR), we only re-rank

---

Probabiltyy
Dropout
CodeBert
GodeBert+GAT
SmBoP
BRIDGE
NatSQL
Answer Iriggering
Precision
0.9
0.9
0.9
0.8
0.7
0.7
0.7
0
200
400
600
800
1000
0
200
400
600
800
1000
250
500
750
1000
Questions Answered
Questions Answered
Questions Answered
1.0
1.0
1.0
Interaction Triggering
Accuracy
0.9
0.9
0.9
0.8
0.7
0.7
0.7
200
400
600
800
1000
0
200
400
600
800
1000
0
250
500
750
1000
Number of Interactions
Number of Interactions
Number Interactions
Figure 2: Performance simulated interactive semantic parsing with three base parsers.
the beams whose top-ranked prediction has a score below given threshold. For simplicity, use decision threshold of 0.5 for error detection.
ger and interaction trigger in interactive semantic parsing systems.
Answer triggering When using error detectors for answer triggering, the interactive semantic pars- ing system restrain from answering the user's ques- tion when an error is detected. The upper half of Figure 2 demonstrates the change of precision when varying the decision threshold. In general, a high threshold (or lower s) reduces the number of questions answered for higher precision. Con- versely, a lower (or higher s) encourages the sys- tem to answer more questions at the cost of making more mistakes.
As shown Table 4, our eтror detectors can im- prove the performance of BRIDGE v2 and NatSQL in both settings without training any re-ranking supervision. Compared with existing re-rankers, model does not need extra pppreeparameter for performance gain, even in the re-ranking-al all set- ting. However, re-ranking hurts the performance of SmBoP. attribute this to the larger train-test dis- crepancy due to the bottom-up nature of SmBoP's decoder. As discussed in Section 3.2 and Section 4.2.2, SmBoP produces more diverse beam predic- tions, but its behavior is less consistent on the test split. While the diversity benefits the quality of data for training error detectors, the inconsistency makes re-ranking on the test split harder. Although SmBoP is the strongest parser among the three, state-of-the-art text-to-SQL. parsers predominantly use autoregressive decoders. Therefore, we still expect our approach to be generally applicable. We further perform 0-shot re-ranking evaluation on the more challenging KaggleDBQA (Lee et al., 2021) dataset (Appendix B). CodeBERT+GAT improves BRIDGE v2's accuracy from 20.5% 21.8%.
Because of the high precision on positive sam- ples, the proposed error detectors outperform both bascline methods and allow the system to answer more questions at higher precision. As shown by Table 5, when maintaining precision of 95%, our error detectors allow the system to answer 76% to 175% more questions compared to parser- dependent metries.
Model
SmBoP
BRIDGE
NatSQL
Probability Dropout
8
24
302
295
104
213
showing good generalization unseen datasets.
CodeBERT
498
247
441
CodeBERT+GAT
520
286
395
4.2.4 Action Triggering in Interactive Systems
Table 5: The number of questions each parser could answer when maintaining precision of 95%,
this section, we evaluate the potential gain of using our error detection model as an answer trig-

---

NatQQ
BRIDGE
SmBoP
Graph
Data Source
Acc
AUC
RR
Acc
AUC
Acc
AUC
NatQQL"
76.3
79.2
71.3
Simplified
in-domain
79.0
83.8
71.2
73.3
76.8
78.5
71.3
Simplified
cross-domain
81.7
86.9
72.5
75.5
81.7
79.9
78.2
Original
cross-domain
81.7
87.2
71.9
75.0
79.9
79.5
76.0
Table 6: Ablation results using the CodeBERT GAT model trained on data collected from NatSQL. We report accuracy, AUC, re-ranking-all all (RR) performance on NatSQL's test split as in-domain evaluation and report accuracy and AUC when tested SmBoP and BRIDGE v2 generalization evaluation. NaISQL is included for reference.
Model
SmBoP
BRIDGE v2
NatSQI.
error detection following the procedure described in Section 3.2. In this way, we collect in-domain parsing errors in the same setting as Bertrand-DD and T5QL. As shown by Table 6, the error detec- tor trained in-domain errors significantly under- performs the one trained on cross-domain errors. The performance of NatSQL deteriorates after re- ranking, which is consistent with the findings of previous re-raskers. Thus, we conclude that collect- ing high-quality parsing errors a realistic cross- domain setting is critical to building an accurate and generalizable error detector.
Probability
535
Dropout
412
566
597
CodeBERT
370
555
421
Cod:BERT+GAT
356
538
384
Table 7: The number of interactions each parser needs with interaction triggering to reach accuracy of 95%.
Interaction triggering We simulate the potential gain of more accurate interaction triggers by assum- ing oracle error correction interactions, where any detected error would be fixed through interactions with users. Ideally, we would want get higher system accuracy with fewer interactions. The lower half of Figure 2 illustrates the change of accuracy at different interaction budgets.
Simplified graph input In this setting, do not simplify constituency parse trees and SQL ab- stract syntax trees when constructing input graphs for graph neural networks. Table 6 shows that the model without graph simplification slightly out- performs the one using simplified graphs in AUC. Despite that, its re-ranking and cross-parser gener- alization performance are lower. We hypothesize that graph simplification could maintain important structural features of the input and improve the model's generalization ability by alleviating over- fitting during training.
Our parser-independen models consistently im- prove upon parser-dependent metrics, resulting in more efficient interactive semantic parsing systems. Due to higher precision and recall on erroneous base predictions, systems using our models could correct more errors and avoid unnecessary interac- tions. As shown by Table 7, depending on the base parser, our model brings a 3.4% to 33% reduction to the number of interactions required for reaching accuracy of 95%.
Conclusion
In this work, we propose the first generalizable parser-independent error detection model for text- to-SQL semantic parsing. Through learning com- positional structures in natural language and SQL queries, the proposed model significantly outper- forms parser-dependent uncertainty metrics and could generalize well to unseen parsers. We further demonstrate the versatility of our approach in error detection, re-ranking, and action triggering through case study with three state-of-the-arr text-to-SQL. parsers featuring different decoding mechanisms.
4.3 Ablation
We perform ablation studies on the impact of cross- domain error collection and graph learning using the CodeBERT+GAT model. We report the models' accuracy, AUC, and re-ranking performance in the re-rank all setting (RR) on the test split of NatSQL We also test the models on BRIDGE v2 and SmBoP to evaluate their generalization ability.
Cross-domain error collection We train Nat- SQL model using the full Spider training set and perform inference on the same data set to get its beam predictions. Then we create training data for
Our experiments highlight the important role of structural features and cross-domain training data in building strong and generalizable error detec-

---

tors for semantic parsing. Potential future work includes (1) developing more advanced architec- tures to better evaluate the semantic correctness of synthesized SQL queries, (2) exploring data synthe- sis strategies to automatically create high-quality training data for error detection models.
Acknowledgements
We would like to thank colleagues from the OSU NLP group for their thoughtful comments. This research was sponsored in part by sponsored re- search award by Cisco Research, NSF IIS- NSF CAREER #1 #1942980, NSF OAC-21 DAC-2112606, and Ohio Supercomputer Center (Center, 1987). The views and conclusions contained herein are those the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Gov- ernment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
Limitations
This work is the first attempt towards building a versatile error detector for text-to-SQL semantic parsing. Although our model is parser-independent, the current data collection process depends on the choice of base parsers. As a result, the collected data may inherit certain biases in the base parsers. Our experiments show that data collected from stronger base parsers helps the model general- ize to weaker parsers. However, how to collect high-quality training data for error detection with stronger base parsers like SmBoP remains an open problem. A promising future direction may be de- veloping a comprehensive data synthesis approach to improve the quality of training data. Grappa (Yu et al., 2021) uses context-free grammar to syn- thesize SQL queries for pre-training Transformer encoders for text-to-SQL parsing. This approach could be adapted to generate syntactically correct but semantically incorrect SQL queries in a con- trollable way.
References
Samuel Arcadinho, David Aparício, Hugo Veiga, and António Alegria. 2022. T5QL: Taming language models for SQL generation.
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019. Global Reasoning over Database Structures for Text- to-SQL. Parsing. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 3659-3664, Hong Kong, China. Association for Computational Linguistics.
Shaked Brody, Uri Alon, and Eran Yahav, 2022. How attentive are graph attention networks? In Interna- rional Conference on Learning Representations.
Another major limitation is that our current model does not consider database schema informa- tion. Since SQL queries are grounded databases, in principle database schema (tables, columns, and foreign-key relationships) should be an important part of error detection. The common practice in text-to-SQL semantic parsing is to linearize the database schema and concatenate all table and col- umn names to the input the Transformer encoder. However, our preliminary experiments show that this operation actually hurts the error detection per- formance. A similar observation is also reported by Kelkar et al. (2020). Nevertheless, our approach performs strongly for eтror detection as it can still effectively capture semantic errors that are free from schema linking mistakes. This can be ex- plained by the high column mention rate in Spider (Pi et al., 2022). Future work could develop more effective entity linking mechanisms to extend our model to more challenging testing environments where schema linking errors are more common.
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao. Su Zhu, and Kai Yu. 2021. LGESQL: Line Graph Enhanced Text-to-SOQ SQL Model with Mixed Local and Non-Local Relations. In Proceedings of the 59th An- nual Meeting of the Association for Computationnl Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 2541-2555, Online, Association for Computational Linguistics.
Ohio Supercomputer Center. 1987. Ohio supercomputer center.
Ziru Chen, Shijie Chen, Michael White, Raymond Mooney, Ali Payani, Jayanth Srinivasa, Yu Su, and Huan Sun. 2023. Text-to-SQL error correction with language models of code. In Proceedings of the ólst Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Shori Papers), pages 1359-1372. Toronto, Canada. Association for Com- putational Linguistics.
Zhangyin Feng, Daya Guo, Dayu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A Pre-Trained Model for Programming and

---

Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Computational Linguisties.
of the Association for Computational Linguistics: EMNLP 2020. pages 48704488888 4888, Online. Association for Computational Linguistics.
Nlya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1030-2042, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Xinyu Pi, Bing Wang. Yan Gao, Jiaqi Guo, Zhoujun Li, and Jian-Guang Lou. 2022. Towards robustness of text-to-SQL models against natural and realistic adversarial table perturbation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 2007-2022, Dublin, Ireland. Association for Compu- tational Linguistics.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-
Guang Lou, Ting Liu, and Dongmei Zhang. 2019. To- wards complex text-to-SQL. in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4524-4535, Florence, Italy, Association for Computational Linguistics.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang. and Zhouhan Lin. 2022. RASAT: Integrating relational structures into pretrained Seq2Seq model for text-to-SQL. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 3215-3229, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Izzeddin Gur, Semih Yavuz. Yu Su, and Xifeng Yan. 2018. DialSQL: Dialogue Based Structured Query Generation. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1339-1349, Mel- bourne, Australia. Association for Computational Linguistics.
Peng Qi. Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Hu- man Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics: System Demonstratiins, pages 101-108, Online. Association for Computational Linguistics.
Magne Jorgensen and Martin Shepperd. 2007. A sys- tematic review of software development cost estima- tion studies. IEFE Transactions on Software Engi- neering, 333-3333....
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj. Saurabh Vaichal, Chandra Khatri, and Peter Relan. 2020. Bertrand-dr: Improving text-to-sql using discrimi- native re-ranker.
Ohad Rubin and Jonathan Berant, 2021. SmBoP: Semi- autoregressive Bottom-up Semantic Parsing. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 31 311-324, Online, Association for Computational Lin- guistics.
Chia-Hsuan Lec, Oleksandr Polozov, and Matthew Richardson. 2021. KaggleDBQA: Realistic evalu- ation of text-Lo-SII. parsers. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 2261-2273 Online. As- sociation for Computational Linguistics.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In AAAI.
Yuntao Li, Bei Chen, Qian Liu, Yan Gao, Jian-Guang Lou, Yan Zhang, and Dongmei Zhang. 2020. "What Do You Mean by That?" A Indeeeeedden In- teractive Approach for Enhancing -SL.. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6913-6922, Online. Association for Computa- tional Linguistics.
Cathrin Weiss, Rahul Premraj. Thomas Zimmermann, and Andreas Zeller. 2007. How long will it take to fix this bug? In Fourth International Workshop on Min- ing Software Repositories (MSR 07:ICSE Workshops 2007), pages 1-1.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging Textual and Tabular Data for Cross- Domain Text-to-SQL Semantic Parsing. In Findings
Ziyu Yao, Yu Su, Huan Sun, and ten-tau Yih. 2019. Model-based Interactive Semantic Parsing: A Uni- fied Framework and A Text-to-SQL Case Study. In

---

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNLP), pages 5447- 5458, Hong Kong, China. Association for Computa- tional Linguistics.
A Fixed Spider Evaluation Script
We fix the following problems in the official evalu- ation process of the Spider dataset.
Incorrect handling of UTF-8 encoded databases. We fix this issue in the Python implementation of the evaluation script.
Ziyu Yao, Yiqi Tang. Wen-tau Yih, Huan Sun, and Yu Su. 2020. An Imitation Game for Learning Semantic Parsers from User Interaction. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6883-6902, Online. Association for Computational Linguistics.
2. Certain examples have gold queries that ex- ecute to an empty result, resulting in false positive labels. For these samples, we use exact set match score instead.
Pengeheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for Gereral-Purpose Code Generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 440-450. Association for Computational Linguistics.
3. Considering the order of returned results when unnecessary. We fix this issue by sorting each column in the execution result in alphabetical order when there is no 'Order By' clause at the top level.
Pengcheng Yin and Graham Neubig. 2019. Reranking for neural semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 4553-4559, Florence. Italy. Association for Computational Linguistics.
4. Duplicated values for queries with 'limit' clause. This results in false negatives for ques- tions querying the maximum or minimum val- ues of a column or aggregation. We fix this issue by removing the limit clause in the gold and predicted SQL when their argument is the same.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-ane -uggeette pre-training for table semantic parsing, In International Conference on Learning Representations.
We show the accuracy of each parser on Spi- der's development set based on the official and our version of evaluation in Table A.1. Although the differences in accuracy are within 1,, the disagree- ments impact 3.3% to 4% of the examples. We use the fixed evaluation script to reduce false posi- tive and false negative labels both for training data collection and evaluation of our model.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Laheled Dataset for Complex and
Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
Parser
Acc
Acc
Disagreement
SmBoP
75.0
75.3
41
Jichuan Zeng, Xi Victoria Lin, Steven C.H. Hoi, Richard Socher, Caiming Xiong, Michael Lyu, and Irwin King. 2020. Photon: A Robust Cross-Domain Text- to-SQL System. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics: System Demonstrations, pages 204-214, Online. Association for Computational Linguistics.
BRIDGE v2
67.3
68.3
34
NatQQ
71.3
71.9
40
Execution accuracy of the three parsers on Spider's development set using official evaluation script (Acc) and our fixed evaluation script (Acc"). Dis- agreement counts the number of samples with different labels.
Table A.I:
Appendices
We provide additional details as follows:
1. Appendix A: Fixed Spider Evaluation Script
2. Appendix D: Modified SQLite Grammar
3. Appendix E: Qualitative Beam Examples

---

Parser
Model
Easy (24.0%)
Medium (43.1%)
Hard (16.8%)
Extra Hard (16.1%)
CodeBERT
85.0/84.9
$1.0/80.5
75.6/81.8
63.3/76.4
SmBoP
CodeBERT+GAT
87.1/86.9
80.5/78.8
80.6/82.7
65.1/76.6
CodeBERT
82.9/81.7
74.6/76.8
75.7/85.7
64.4/75.1
BRIDGE
CodeBERT +GAT
82.7/82.4
76.4/80.8
79.0/86.3
70.0/76.7
CodeBERT
89.0/91.9
82.6/84.6
78.5/84.6 78.9/87.3
72.1/78.5
NatQQL
CodeBERT+GAT
88.7/91.6
81,4/84.0
74.1/80.1
Table A.2: Error detection performance Acc/AUC break down by difficulty the Spider dev set. Difficulty is decided by the official Spider evaluation seript. The proportion of each difficulty type is in the parenthesis.
B KaggleDBQA Results
compare the performance of CodeBERT and Code- BERT+GAT. While CodeBERT can perform better on easy and medium questions for some parsers, CodeBERT+GAT consistently wins on hard and extra hard questions, showing the effectiveness of introducing structural features for harder questions. Since 63. % of the questions are of easy or medium difficulty, the overall evaluation in Table 2 favors CodeBERT.
Re-ranker
CodeBERT+GAT
20.5
RR ED RR
21.8
21.4
Beam Hit
25.9
Table B.3: Execution accuracy of BRIDGE v2 and re- ranking using the CodeBERT + GAT model (left) and CodeBERT (right) on KaggleDBQA, RR: Re-ranking all beams; ED+RR: Re-ranking heams after error detec- tion.
D Modified SQLite Grammar
we use modified version based on a publicly available context-free grammar for SQLite https://githhb.com tree//masterr/s///slliic We slightly modify the publicly available SQLite grammar for Antlr4 to introduce more hierarchical structures of SQL queries at the top level.
To test the generalization abilities of the pro- posed error detecor, we perform 0-shot evaluations on the 370 test examples* in KaggleDBQA( et al., 2021). KagglcDBQA features more realistic database naming and makes entity linking signif- icantly more challenging than Spider. We only experiment with BRIDGE v2 for the following rea- sons: (1) Under the O-shot testing, SmBoP trained Spider got accuracy of 1.6% both for top- and beam hit, making re-ranking meaningless. This is partly due to the failure of its entity linking modules based on span extraction from questions. (2) At the time of writing, the SQL-to-NNaSQL, part of NatSQL's preprocessing code has not been re- leased and its current codebase does not support KaggleDBQA.
We represent SQL keyword 'GROUP BY' by single terminal GROUP __________________ and 'ORDER BY' by ORDER_BY_. The original grammar reuses BY_ for 'BY', which we think breaks the entirety of these two keywords.
Terminals
Non-terminals
We first remove values_clause and rules related to window functions, as they are not used by SQL queries in the Spider dataset. Then we break the select_core non- terminal, which represents SQL query starting with SELECT, into composition of multiple terminals, one for each SQL clause
We present the (-shot re-ranking results with BRIDGE v2 using CodeBERT+GAT in Table B.3. Without any training data, CodeBERT+GAT im- proves BRIDGE v2's accuracy by 1.3%.
Our new SQLite grammar listed as follows:
select_core:
SELECT_ (DISTINCT_ ALL.)? result_clause
C Performance by Difficulty
(from_clause).
In Table A.2, break down the error detection performance in Table by question difficulty and
(group_by_clausee)
*Wo use all examples in examples/* htpollessls_tet http:/*tttst.jooooo in https://githhn www/chi.cccccccecc..c...cc.....a httm//hhaasuanttam 165gaasss
http://wb...//ttt//gaaarrerrraee master/sql/sqlite

---

result_ result_column
(COMMA result_column)
:
FROM_ table_or_subquer,
(COMMA table_or_ _tt.urrssuuuur))
FROM_ join_clause
where_cla clause: WHERE_ expr
coup_by-clausee GROUP BY_ expr (COMMA expr) (HAVING_ expr)?
The original grammar for select_core:
select_core:
SELECT_ (DISTINCT_ | ALL_)? result_column
(COMMA result_column)
(FROM_
Stable_or_subquee
(COMMA
join_clause
)? (COMMA groupByExprtsexpr)>
(WHERE_ whereExpr=expr)?
(GROUP_ BY_ groupByExpr
(HAVING_ havingExpr=expr)
Notice the excessive use of & in the original grammar that fails to represent the hierarchical rela- tionship between the SELECT statement and each clause.
E Qualitative Beam Examples
As mentioned in Section 3.2, the three text-to- SQL parsers behave differently. We present their beam predictions on two samples in our training split in Table E.4. We can observe that SmBoP and NatSQL could generate more executable SQL queries than BRIDGE v2. Both SmBoP and Nat- SQL are capable of generating diverse errors, but NatSQL's beam predictions are more likely to share prefixes. As an example, SmBoP generates diverse SELECT clauses on both samples, while the SELECT clauses predicted by BRIDGE v2 and NatSQL do not change.

---

Question:
How many heads of the departments are older than 56?
Gold SQL:
SELECT COUNT (*) FROM head WHERE head. age > 56
SELECT COUNT (*) FROM head WHERE head. aga > 56
SELECT head. name FROM head WHERE head. age > 56
SmBoP
SELECT MAX age) FROM head WHERE head.ape laad 56
SELECT aga FROM head WHERE head.age > 56
SELECT FROM head WHERE head.age 56
BRIDGE v2
SELECT COUNT (*) FROM head WHERE head.age age
SELECT (O)NT(*) FROM head WHERE head. age 56
SELECT COUNT(*) FROM department WHERE department department_iu in (SELECT
NatSQL
management. departmentt_D FROM management, head WHERE head, age
SELECT COUNT (*) FROM head WHERE head.age agd SELECT COUNT FROM head WHERE head.age 56 SELECT COUNT (+) FROM head WHERE head. age
Question:
Show the names of the three recent festivals.
Gold SQL:
SELECT aestival_detall. weetalffettilllaaaaa FROM festival_detail ORDER BY festival_detail,
DESC LIMIT 3
SELECT
festival_name FROM festival_detar ORDER BY festival_detail. year
DESC LIMIT 3
SELECT festival detiail festtiilleet festival_ name FROM cestival_detaiie WHERE festival.detail.yea.
(SELECT MAX( festival_detail hatpaaldeellllll..... year ) FROM restiva_dettall
SELECT 3 FROM festival_deter _el_dddealllll WHERE festival_detail year = (SELECT MAX
SmBoP
destiva__ettal.yeee SELECT MAX( festival_detail.year FROM festival_detail ORDER BY festival_detail year
FROM festival_detail)
DESC LIMIT 3 SELECT MAX( festival_detail.yy. FROM festival_detail ORDER BY festival_detail.
DESC
SELECT Fetiial..aaall..eeiillllllll FROM festival_detail ORDER BY festival_detail Year
ASC LIMIT
SELECT FetiiialddtaaalFesttsttt
FROM festival_detail ORDER BY festival_detail. Year
BRIDGE v2
LIMIT 3
SELECT festival_ Festolda FROM festival_detail ORDER BY festival_detail. Year
DESC LIMIT 3
SELECT festival_detan festivallfestivll_aam. FROM festival_detail ORDER festival_detail.yye
DESC LIMIT 3 SELECT
FROM festival.detail ORDER BY festival
ASC LIMIT SELECT festival_detail, festivallaaivvllanee
festival_detail FROM festival_detail ORDER
NatSQL
BY destival_detaillyyee DESC LIMIT SELECT httplldddtalllffiiiieee
FROM festival_detail
SELECT Testival_detail festival_name _ame FROM festival _ettail
GROUP
BY
festival_detail festival_nane ORDER BY dettival.detal..vee DESC LIMIT
Table E,4: Example beam predictions collected from three base parsers in training dataset.


Reviews:
[{"review_id": "GigZscYFvG", "reviewer": "Reviewer_12YA", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort that the reviewer put into reviewing our submission.\n\n**[The idea is not novel]**\n\nThe contribution of this work is not limited to the design and implementation of the proposed error detection model. Besides building the **first parser-independent error detection model for text-to-SQL parsing**, which the reviewer finds to have \"obtained good performance in multiple tasks,” our work is also the **first comprehensive study on error detection for text-to-SQL parsing**. This includes task formulation, evaluation settings, and performance comparison between several error detection methods. \n\n**[This error detection model is independent with parsers, it need to be trained, and it need to be called during inference]**\n\nTraining a parser-independent error detection model like ours has a few advantages: (1) It can be applied to black-box semantic parsers (2) It can generalize well to unseen parsers. In our experiments, we compared against alternative methods that are parser-dependent and do not need to be trained, such as probability and drop-out based confidence measures, and showed that our proposed method can perform better.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The error detection model proposed by this paper obtained good performance in multiple tasks, i.e., error detection, SQL re-ranking and action triggering. Meanwhile, this error detection model is independent with parsers.", "reasons_to_reject": "1. The idea is not novel, this error detection model is just a semantic matching model between natural question and SQL query.\n2.  This error detection model is independent with parsers, it need to be trained, and it need to be called during inference.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a parser-independent error detection model for text-to-SQL semantic parsing task. This error detection model utilizes the compositional structures in natural language and the corresponding SQL queries, and it outperforms other parser-dependent models and could generalize to unseen parsers. \n\nMeanwhile, the author demonstrated that this error detection obtained gains in several tasks, such as error detection, SQL re-ranking and action triggering.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bZJy18c0d6", "reviewer": "Reviewer_nSMo", "summary": "", "questions": "How is the effect of beam size during data generation on the quality of the error detection model?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that the reviewer finds our cross-domain data generation method “interesting and effective” and our experiment results “promising.” We also thank the reviewer for the valuable constructive comments.\n\n**[The chosen base models are not strong enough]**\n\nWe agree with the reviewer that our error detection model should work for  stronger parsers as well. In this work, we prioritized covering different decoder architectures and picked strong parsers at the time of our experiments. To address this concern, now we report new results on RESDSQL + NatSQL[1] , which is the strongest open-source parser and ranks 5th on the Spider leaderboard (as of 8/28/23). Due to time and computing restraints, we test the RESDSQL + T5-large + NatSQL variant using the released checkpoint [2], which is still stronger than SmBoP (81.9% vs 75.0% EX on Spider dev as reported by their papers). Since RESDSQL uses a sequence-to-sequence decoder, we compare against using its prediction probability as the uncertainty baseline. We follow the same cross-parser evaluation setup in Section 4.2.2 and test the CodeBERT + GAT model trained on data collected from NatSQL. The results are in the following table (+ and - denotes positive and negative examples):\n\n|Parser | +Precision | +Recall | +F1 |-Precision| -Recall | -F1 | Acc | AUC |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\nRESDSQL$^p$          |**82.8**|91.7|**87.0**|50.3|31.3|38.4|78.8|79.4|\nCodeBERT + GAT (ours)|82.7|**92.5**|**87.0**|**54.3**|**32.9**|**38.7**|**78.9**|**81.0**|\n\n\nOur model performs comparably or better than RESDSQL$^p$ on all metrics, which is consistent with our cross-parser generalization results (Table 3). The improvement is most salient under the AUC metric, which is our main metric. **Notably, this is generalizing from a weak parser to a stronger one.** We can reasonably expect our method to perform even better with data collected from RESDSQL itself, which we will do in the revised version.\n\n**[It is unclear whether the conclusion in the paper applies to in-domain evaluation as well]**\n\nWe agree that error detection for in-domain settings is also an important aspect. In this work, we focus on the Spider dataset because it has been considered more challenging than text-to-SQL datasets with in-domain evaluation settings (e.g., single-database datasets such as Geoquery [4]), due to its cross-domain setup and more complicated SQL structures. In light of the promising performance of our approach on this more challenging dataset, we are optimistic about adapting our method to in-domain evaluation settings.\n\n**[The effect of beam size during data generation]**\n\nThere are two factors we take into consideration when generating our training data. (1) It is important to collect diverse executable errors. This is shown by the statistics of generated data (Table 1) and results of cross-parser generalization (Sec. 4.2.2 and Table 3). (2) The collected data should have the same error distribution as the parser applied at test time. We use the same hyperparameter as they are in the official implementation of each parser during data collection\n\nHence, changing beam size during beam search may have a mixed influence on data quality. Intuitively, using a larger beam size would help to generate more diverse data that has broader coverage of different types of errors. However, it could also deviate from the real error distribution in testing. As discussed in the Limitations section, future work could explore more principled data synthesis approaches.\n\n**[How the dependency parse trees and constituency trees are merged]**\n\nSince edges in a dependency parse tree are between two actual tokens in a text snippet, which are leaf nodes in a constituency tree, we directly add these edges to the constituency parse tree. We will add more detailed descriptions to clarify this in the revised version.\n\n**[The use of “hit”, “miss”, and “zero-shot” are confusing]**\n\nWe apologize for the confusion caused by these terms. The use of “hit” and “miss” are borrowed from the notion of beam hit and beam miss in beam search used in previous work such as Bertrand-DR [3]. Your understanding of “zero-shot” is correct. We will remove the term “zero-shot” in Section 4.2.2. Thanks for pointing these out and we will improve the clarity of the writing in the revision.\n\nReferences:\n\n[1] Li, Haoyang, Jing Zhang, Cuiping Li, and Hong Chen. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, pp. 13067-13075. 2023.\n\n[2] https://github.com/RUCKBReasoning/RESDSQL#step1-prepare-checkpoints\n\n[3] Kelkar, Amol, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, Chandra Khatri, and Peter Relan. \"Bertrand-dr: Improving text-to-sql using a discriminative re-ranker.\" arXiv preprint arXiv:2002.00557 (2020).\n\n[4] Zelle, John M., and Raymond J. Mooney. \"Learning to parse database queries using inductive logic programming.\" In Proceedings of the national conference on artificial intelligence, pp. 1050-1055. 1996.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides an interesting recipe for training error detection models. While the particular model architecture is not very novel, the additional of graph attention network is not critical to the performance, I find the cross-domain data generation method to be interesting and it proves to the most effective in the ablation studies.\n- The paper shows promising results on error detection, simulated interactive semantic parsing, and reranking on a combination of base parser and experiment settings.", "reasons_to_reject": "- The chosen base models are not strong enough. According the Spider leaderboard, the three models used in the paper are all below 15th rank. There are quite many options in the leaderboard with code available. It would be really informative to see whether the proposed model works for these stronger semantic parsers or not.  [This concern is addressed by the authors' responses below.]\n- It is not clear how generalizable is the cross-domain data generation method to other datasets and settings. The Spider benchmark is set up specifically with a large variety of databases and the database split aims specifically at measuring cross-domain generalization. It is not unclear how whether the conclusion in the paper applies to in-domain evaluation as well or not.\n- The reranking result with SmBoP is worse than not using reranking at all. This suggests the proposed model might not work so well with strong models (SmBoP is the strongest base model out of the three tested). Also see the first point in this section.", "questions_to_authors": "How is the effect of beam size during data generation on the quality of the error detection model?", "paper_topic_and_main_contributions": "This paper considers the task of error detection for text-to-SQL semantic parsing: given a prediction result, the model needs to decide whether the prediction is correct or not. This type of error detection models can be used for quality control (ensuring X% precision by filtering out bad predictions) and for reranking.\n\nThe error detection model is based on CodeBERT and graph attention networks. The graphs used are obtained through parse trees (dependency and constituency) on the utterance side, and abstract syntax trees on the SQL output side. Additionally, the paper proposes to train the error detection model with cross-domain data, which are generated by training the parsers on a subset of domains and running inference on a different subset of domains. (Here domains = databases in the Spider dataset).\n\nExperiment results show that the proposed error detection model consistently outperforms using \"confidence\" measures directly from the three baseline parsers, SmBoP, BRIDGE v2, and NatSQL on error detection task. The paper also shows improved execution accuracy through reranking for BRIDGE v2 and NatSQL parsers.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is not clear how the dependency parse trees and constituency trees are merged (line 260).\n\nThe terms \"hit\" and \"miss\" are not explained in the paper, and it took me a while to figure out what they mean.\n\nThe terminology of \"zero-shot\" in section 4.2.2 is misleading. The error detection model is indeed trained on a different parser, but the training data and databases are seen if I understand correctly."}}, {"review_id": "qZYUiMLQY3", "reviewer": "Reviewer_jWP3", "summary": "", "questions": "* line 267: can you provide an example of the removed information? why is it that they “do not carry much semantic information\"?\n* have you considered training the proposed model on the output of more than one parser (e.g. BRIDGE+NatSQL) and test on the other (e.g. SmBoP)? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s effort in reviewing our submission and are grateful for their recognition of our work.\n\n**[ln267: Example of the removed information and why]**\n\nWe remove two kinds of information during preprocessing: foreign key relations  (i.e. ON tab1.col1 = tab2.col2) and related keywords (i.e. JOIN and AS) in FROM clauses. Since foreign key relations are specific to the underlying database, it may not correspond to the semantics of the natural language question. In addition, foreign key relations usually consist of a lot of words, especially when joining more than two tables in the FROM clause. This could make the model prone to overfitting, as shown by our ablation studies (Table 6). Therefore, we only keep the table names in the simplified SQL query.\n\nAn example from the Spider dev set:\n\nQuestion:\n\nList all singer names in concerts in year 2014.\n\nOriginal SQL:\n\n```sql\nSELECT T2.name FROM singer_in_concert AS T1 JOIN singer AS T2 ON T1.singer_id  =  T2.singer_id JOIN concert AS T3 ON T1.concert_id  =  T3.concert_id WHERE T3.year  =  2014\n```\n\nSimplified SQL:\n\n```sql\nSELECT singer.name FROM singer_in_concert singer concert WHERE concert.year  =  2014\n```\n\nIn this example, the original SQL has 26 words (segmented by whitespace) and the simplified SQL only has 10 words. Despite the massive difference in length, they convey the same semantics.\n\n**[Training the proposed model on the output of more than one parser]**\n\nWe thank the reviewer for this suggestion and agree that mixing data from multiple parsers could potentially enhance the performance and generalization ability of error detection models. We did not include this setting in the paper to keep our experiment settings succinct.\n\nTo address this question, we test the suggested setting by merging the training data collected from BRIDGE v2 and NatSQL. We train a CodeBERT + GAT model and test it on SmBoP, following the same cross-parser evaluation protocol as in Section 4.2.2.  The new results are presented in the last row of the following table (+ and - denotes positive and negative examples). The first 3 rows are taken from Table 3 in our paper.\n\n| Parser | Source |+Precision|+Recall|+F1|-Precision|-Recall|-F1|Acc|AUC|\n|:-|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\nSmBoP$^s$|/                      |81.5|91.9|85.7|**56.6**|25.3|29.4|76.9|79.2|\nCodeBERT + GAT|BRIDGE v2         |80.0|**95.9**|86.8|44.1|14.4|18.8|77.8|77.0|\nCodeBERT + GAT|NatSQL            |83.1|93.3|**87.6**|56.0|31.5|38.1|**79.9**|78.2|\nCodeBERT + GAT|BRIDGE v2 + NatSQL|**83.6**|92.4|87.4|55.5|**35.5**|**41.4**|79.8|**80.5**|\n\n\nMixing training data generated by different parsers does enhance the coverage of errors and **improves the error detection model’s performance on negative test examples**, shown by the sizable improvement in negative recall and F1, **as well as overall performance** shown by the increase of AUC.\n \n\n**[Typos Grammar Style And Presentation Improvements]**\n\nWe thank the reviewer for pointing out this issue. We will correct it in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Given the recent success of text-to-SQL parsers, the interest and importance of the task of error detection in the output of text-to-SQL systems is increasing in the community.\n* The proposed method reports strong experimental results. Moreover, the paper includes extensive experimentation on a number of scenarios that are relevant to error detection and handling, using a number of parsers.\n* The experiment described on section 4.2.4 (Action Triggering in Interactive Systems) is a strong point of the paper; it views the error detection problem from a more abstract focus point and considers the practical implications of parser errors.\n* The paper is very well written and easy to follow.", "reasons_to_reject": "n/a", "questions_to_authors": "* line 267: can you provide an example of the removed information? why is it that they “do not carry much semantic information\"?\n* have you considered training the proposed model on the output of more than one parser (e.g. BRIDGE+NatSQL) and test on the other (e.g. SmBoP)? ", "paper_topic_and_main_contributions": "The paper presents a semantic error detection approach for the output of text-to-SQL semantic parsers, consisting of (a) a method to generate a cross-domain error detection dataset, (b) a model for the error detection problem, and (c) a comprehensive study on error detection for text-to-SQL semantic parsing.\nSpecifically, the authors collect a dataset of errors using weak versions (trained on smaller datasets from fewer databases than their default version) of base parsers.\nThe proposed model uses an encoder (based on CodeBERT), whose input is the natural language questions and generated SQL queries. The output of the model is the aggregation of the output of the encoder and that of two graph models, one operating on natural language information (using dependency and constituency trees) and one operating on SQL information (using query abstract syntax trees). The model is fine-tuned with cross-entropy loss, where positive examples are gold (question, SQL) tuples, and negative examples come from the collected dataset of errors.\nThe authors report results on error detection, comparing their method with simple prediction probability thresholds and dropout-based uncertainty, showing that the proposed method outperforms the baselines. Moreover, the paper includes experiments on cross-parser generalisation, reranking of beam results, and action triggering in interactive systems, using three different parsers with different decoding implementations. \nInterestingly, the proposed method is not tied to a specific parser implementation, and can be used for the output of a number of parsers.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* line 102: it might be a good idea to start with the phrase “Most existing neural text-to-SQL parsers” instead of “Existing text-to-SQL parsers”, since there is a number of (mainly older) approaches that do not follow any of the three decoding mechanisms mentioned."}}]

Author rebuttal:

Meta-review: The authors propose a re-ranker model for text-to-SQL generations. The model is build on top of CodeBERT with extra GNN on top.

Reviewers have different views for this paper:
Reviewers 12YA and nSMo recognize the positive results presented in the paper, but state that such the concept of a re-ranker is well explored and that the baseline parsers on top of which the re-ranker is applied are not the best text-to-SQL models out there.
I tend to agree that it is not clear how this work is different than past work that presents a re-ranker -- the concept is not new, and was specifically proposed for semantic parsing in the past. I also believe the following reference is missing -- "Reranking for Neural Semantic Parsing". I would expect stronger baseline parsers and comparison to past re-rankers.

On the other hand, reviewer jWP3 recognized the positive results and the extensive experiments, and sees no issues with the paper.
Decision: Accept-Findings