Venue: ACL 2023

# Text-to-SQL Error Correction with Language Models of Code

Ziru Chen\({}^{1}\), Shijie Chen\({}^{1}\), Michael White\({}^{1}\), Raymond Mooney\({}^{2}\) Ali Payani\({}^{3}\), Jayanth Srinivasa\({}^{3}\), Yu Su\({}^{1}\), Huan Sun\({}^{1}\)

###### Abstract

Despite recent progress in text-to-SQL parsing, current semantic parsers are still not accurate enough for practical use. In this paper, we investigate how to build automatic text-to-SQL error correction models. Noticing that token-level edits are out of context and sometimes ambiguous, we propose building clause-level edit models instead. Besides, while most language models of code are not specifically pre-trained for SQL, they know common data structures and their operations in programming languages such as Python. Thus, we propose a novel representation for SQL queries and their edits that adheres more closely to the pre-training corpora of language models of code. Our error correction model improves the exact set match accuracy of different parsers by \(2.4\text{--}6.5\) and obtains up to \(4.3\) point absolute improvement over two strong baselines.1

Footnote 1: Our code and data are available at [https://github.com/OSU-NLP-Group/Auto-SQL-Correction](https://github.com/OSU-NLP-Group/Auto-SQL-Correction).

0
Footnote 1: The Ohio State University

\({}^{2}\)The University of Texas at Austin \({}^{3}\)Cisco Research

{chen.8336, chen.10216, white.1240, su.809, sun.397}@osu.edu mooney@cs.utexas.edu {apayani, jasriniv}@cisco

## 1 Introduction

Text-to-SQL parsing is a classic semantic parsing task that finds wide applications Zelle and Mooney (1996); Tang and Mooney (2000). Since the release of Spider Yu et al. (2018), a cross-database text-to-SQL benchmark, many semantic parsers with decent performance have been developed Lin et al. (2020); Wang et al. (2020); Deng et al. (2021); Rubin and Berant (2021); Scholak et al. (2021). Nonetheless, state-of-the-art semantic parsers are still not accurate enough. As a result, their users need to constantly correct wrongly predicted SQL queries, which can be as time-consuming and errorprone as writing a SQL query from scratch Jorgensen and Shepperd (2007); Weiss et al. (2007). Therefore, in this paper, we study the problem of automatic text-to-SQL error correction to better assist users in querying complex databases.

We first highlight that it is essential to factor in the _compositional substructures_ within SQL queries, such as abstract syntax trees Yin and Neubig (2017); Guo et al. (2022) and data-flow graphs Guo et al. (2021), instead of treating code snippets as string sequences. Compared to individual tokens, substructures (e.g. SQL clauses) include more context of the entire program and are more semantically meaningful. Consequently, edit patterns of such substructures are more intuitive for humans to understand and easier for language models to learn. Moreover, while the pre-training corpora for language models of code, such as CodeT5 Wang et al. (2021), do not include many SQL queries based on their documentation, they naturally contain _abundant examples of common data structures_ like dictionaries. Therefore, we hypothesize that transforming unfamiliar SQL queries into familiar data structures can help language models of code better perform structural editing of SQL queries.

Based on these observations, we develop our error correction model and make two contributions. First, we propose considering SQL clauses instead of tokens as basic semantic units for editing. Using a context-free grammar, we can decompose a SQL query and identify its clauses by traversing its abstract syntax tree. Second, we propose a new representation of SQL queries and their edits that adheres more closely to common code pre-training corpora, including CodeSearchNet Husain et al. (2020), and makes the structures of a SQL query more explicit. With a decomposed SQL query, we pair each clause with its SQL keyword and represent the entire query as a Python dictionary. Then, we format edits on a wrong SQL query as a program that modifies data of the query's corresponding dictionary. Unlike token-level edits in existing work Zhang et al. (2023), such dictionary operations define all edits unambiguously and can be directly executed with a Python interpreter.

Through comprehensive experiments with different representations, we show that: (1) our proposed representation has the lowest zero-shot perplexity

with CodeT5; (2) simply changing token-level edits to clause-level edits can effectively improve the performance of our models; and (3) our method improves the exact set match accuracy of different parsers by \(2.4\text{--}6.5\) and obtains up to \(4.3\) point absolute improvement over two strong baselines.

## 2 Text-to-SQL Error Correction

Given a natural language utterance \(\mathbf{u}\), a database schema \(\mathbf{s}\), and a wrong SQL query \(\mathbf{q_{-}}\) produced by an existing parser, our goal is to develop an error correction model that predicts a sequence of edit actions \(\mathbf{e}\) and the correct query \(\mathbf{q_{+}}\). Following previous work Zhang et al. (2023), we formulate our task as sequence-to-sequence generation:

\[P(\mathbf{y}|\mathbf{x})=\Pi^{T}_{t=1}P(\mathbf{y}_{t}|\mathbf{x},\mathbf{y}_ {1:t-1}) \tag{1}\]

where \(\mathbf{x}=[\mathbf{u};\mathbf{s};\mathbf{q_{-}}]\) is the concatenation of the given inputs and \(\mathbf{y}=[\mathbf{e};\mathbf{q_{+}}]\) is the concatenation of all edit actions and the resulting correct query. In this section, we study different representations of SQL queries (Section 2.1) and edits (Section 2.2) to better leverage language models of code.

### Query Representation

We consider two representations for a predicted query: (1) the original SQL format and (2) our proposed PyDict (Python Dictionary) representation. To prepare for editing, we disambiguate each SQL query following Rubin and Berant (2021), including lower-casing non-value tokens, resolving table references, and formatting punctuation. This preprocessing normalizes SQL queries predicted by different base parsers and the gold annotations into the same format. To build our PyDict representation, we parse a SQL query into its abstract syntax tree (AST) with Spider's context-free grammar. We use depth-first search to traverse through the AST, find any nested substructures, and construct the dictionary representation bottom-up. Table 1 shows the "SQL" and "PyDict" representations of a SQL query (more details in Appendix A).

### Edit Representation

We first follow Zhang et al. (2023) to use token-level edit representation with special tokens (Table 1), which have unique entries in the tokenizer and the model's embedding layer to describe _Replace, Insert_, and _Delete_ edit actions (more examples in Appendix F). However, we realize this representation can sometimes be ambiguous. As shown in Table 1, the span "twects.text" appears twice in the SQL query. This repetition would confuse the error correction model with which span to replace when generating the corrected query. Also, the ambiguity makes it difficult to implement rules and directly carry out the edit actions on the wrong query. Hence, we change the token-level edit representation to clause-level, which includes more context of the query to make different edits more distinguishable. In our experiments (Section 4.1), we demonstrate that this simple modification is already effective. Our program representation further improves the performance because it is more similar to the code pre-training corpora and eliminates the need to learn special tokens' representations.

## 3 Experimental Setup

### Data Synthesis for SQL Error Correction

To train a text-to-SQL error correction model, we need to collect a set of wrong SQL parses that reflects a realistic distribution of errors (Section 4.2) as our training data. We synthesize this dataset by

\begin{table}
\begin{tabular}{l l c} \hline \hline  & **Query Representation** & \multicolumn{2}{c}{**Edit Representation**} \\ \cline{2-3}
**SQL** & select tweets.text & \multirow{2}{*}{**Token-Level**} & \multirow{2}{*}{**tokens.text**} & \multirow{2}{*}{**<ReplaceNew>**} \\  & from tweets & & \\  & order by **twects.text** & & \\ \hline \multirow{3}{*}{**PyDict**} & sqI = \{ & **CloseOld**\textgreatergreatergreatergreatergreatergreatergreatergreatergreatergreatergreatergreatergreatergreatergreater “from”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”orderBy”: ”\(\}\}\end{array}\) & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Example representations for a wrong SQL query and the _Replace_ edit action. The corresponding natural language utterance is “List the text of all tweets in the order of date.” For token-level and clause-level representations, we format them as “<ReplaceOld¿ Span of wrong tokens/clauses ¡ReplaceEnd¿ Span of correct tokens/clauses ¡ReplaceEnd¿”, where ¡ReplaceOld¿ ¡ReplaceNew¿ Span of correct tokens/clauses ¡ReplaceEnd¿”, where ¡ReplaceOld¿ ¡ReplaceNew¿ Span of correct tokens/clauses ¡ReplaceEnd¿ Span of correct tokens/clauses ¡ReplaceEnd¿ Span of correct tokens/clauses ¡ReplaceEnd¿ Span of correct tokens/clauses/clauses/clause/clauses/cl 

performing 5-fold cross-validation on each parser, which approximates the actual evaluation setting.

Following the evaluation setup in Yu et al. (2018), we split Spider's training set into five roughly equal subsets by different databases. For each cross-validation fold, we train a text-to-SQL parser (Section 3.2) on four subsets and evaluate it on the remaining one. At inference time, we perform beam search with size 20 for each example and collect grammatical and executable parses in the beam.2 If a SQL parse is not an exact set match or execution match to the gold annotation, we label it wrong and include it in our training set for error correction. Having synthesized our training dataset, we randomly sample 8 databases and their associated questions to construct a held-out development set. For development set examples, we only keep incorrect SQL parses with the highest beam confidence. For our error correction test set, we train each parser on the _full_ Spider training set and evaluate it on the original Spider's development set without modifications. We similarly keep SQL parses with exact match or execution match errors. Table 2 summarizes the statistics of our data.

Footnote 2: Due to SmBoPs’s bottom-up decoding, we keep its original beam size and collect the top-20 unique beam predictions.

### Models

Text-to-SQL base parsers.We choose three text-to-SQL parsers with different decoding strategies and levels of performance (Table 3). We elaborate on our selection criteria in Appendix B.

* **CodeT5**Wang et al. (2021): We fine-tune CodeT5-base following Xie et al. (2022). This parser represents those using beam search decoding and having a lower accuracy.
* **BRIDGEv2**Lin et al. (2020): A representative parser with constrained decoding and achieving a medium-level accuracy.
* **SmBoP**Rubin and Berant (2021): A representative parser with bottom-up decoding and achieving higher accuracy.

Error correction models.We use two language models of code in all our experiments:

* **CoditT5**Zhang et al. (2023): A language model pre-trained for code editing tasks by injecting noises to code snippets in CodeSearchNet Husain et al. (2020) and then denoising with token-level edit representations.
* **CodeT5**Wang et al. (2021): A language model pre-trained for general code understanding and generation with four different pre-training objectives.

We compare the existing SQL+Token-Level representation with our proposed ones: SQL+Clause-Level, PyDict+Clause-Level, and PyDict+Program on CodeT5 and the first three on CoditT5.3 Implementation details are in Appendix C.

Footnote 3: We did not use CoditT5 for PyDict+Program because it was pre-trained on token-level edit representations. Its decoder may be specialized in generating edits instead of programs.

### Evaluation

We use the increase in Exact Set Match (EM) and Execution Match (EX) accuracy on our error correction test set to measure each model's performance. Because CoditT5's experiments assume the input program has at least one error, we keep this assumption for fair comparisons. To construct a test set satisfying this assumption, we have to compare parser-generated SQL queries with gold annotations (Section 3.1). Thus, we use the Spider development set as our test set and split the Spider training set to build a held-out development set (Table 2) to select model checkpoints during training. We also include results on our held-out development set in the appendix (Table E.1).

## 4 Results and Analysis

### Main Results

We summarize our main results in this section. To ensure robustness, we repeat all experiments with 3 different random seeds and report the average performances with standard deviations. Our model can also be used in an interactive framework that allows users to select edit actions from the top-\(k\) beam candidates. We include more experiments with simulated user interactions in Appendix E.

Our representation's perplexity is the smallest.We validate that our PyDict+Program representation adheres more closely to the code pre-training corpora by measuring its zero-shot perplexity on CodeT5 using our development set (Section 3.1).

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **CodeT5** & **BRIDGEv2** & **SmBoP** \\ \hline \# of Train & 47,020 & 24,776 & 20,083 \\ \# of Dev & 448 & 448 & 448 \\ \# of Test & 430 & 392 & 310 \\ \hline Avg. Train Edits & 2.34 & 3.11 & 2.72 \\ Avg. Dev Edits & 2.70 & 3.29 & 3.31 \\ Avg. Test Edits & 1.84 & 1.51 & 1.47 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of data statistics.

 

As shown in Figure 1, by representing data in PyDict, we can reduce the perplexity of CodeT5 by 2 orders of magnitude. After augmenting it with our program representation, we further reduce the zero-shot perplexity of CodeT5 to only \(5.96\times 10^{2}\), 3 orders of magnitude less than the SQL+Token-Level representation (\(1.26\times 10^{5}\)).

Clause-level editing is more effective, especially when represented in PyDict+Program.Since CodeT5 consistently outperforms CoditT5 with the same representations, we focus on comparisons among CodeT5 variations. As shown in Table 3, compared to CodeT5-SQL+Token-Level, only CodeT5-PyDict+Program achieves statistically significant improvement on all three parsers, while clause-level models fail McNemar's significance test for some parsers. More concretely, it achieves up to 4.3 point more absolute improvement on EM accuracy (\(68.2\to 72.5\); BRIDGEv2) and 3.7 point more absolute improvement on EX accuracy (\(69.4\to 73.1\); BRIDGEv2). Overall, CodeT5-PyDict+Program can boost the parsers' EM accuracy by 2.4-6.5. Thus, both clause-level editing and PyDict+Program representation can better take advantage of language models of code.

### Error Analysis

Additionally, we conduct an error analysis (Table 4) by sampling 100 wrong parses from all three parsers and classifying them into five categories:

* _Database Grounding_: A generated SQL query has the correct structure, but some table/column names or entity values are wrong.
* _Incorrect Structure_: A generated SQL query has missing, wrong, or redundant structures.
* _Syntax & Grammar_: A generated SQL query violates the programming language's syntax.
* _False Negative_: A generated SQL query is semantically correct but not captured by evaluation metrics, or the gold annotation is wrong.
* _Other_: All other errors, such as wrong aggregation functions, besides the above categories.

Since the error distributions for each parser are similar, as an example, we discuss our findings based on the strongest parser, SmBoP:

**Database grounding is the major type of error.** Among the 100 samples from SmBoP, we find that 54 of them have database grounding errors. Particularly, SmBoP predicts wrong table/column names in 34 parses, inaccurate entity values in 9 parses, and incorrect JOIN relations in 11 parses. Our CodeT5-PyDict+Program model can successfully fix 16 of the 54 erroneous parses, including 10 parses with wrong table/column names, 4 parses with inaccurate entity values, and 2 parses with incorrect JOIN relations. We hypothesize that

Figure 1: CodeT5’s zero-shot perplexity (in log scale) of all four representations on our synthesized SQL error development set.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**Query**} & \multirow{2}{*}{**Edit**} & \multicolumn{2}{c}{**CodeT5**} & \multicolumn{2}{c}{**BRIDGEv2**} & \multicolumn{2}{c}{**SmBoP**} \\ \cline{3-8}  & & & EM & EX & EM & EX & EM & EX \\ \hline No Edit & N/A & N/A & 62.7 (-) & 63.6 (-) & 70.1 (-) & 68.2 (-) & 74.6 (-) & 75.3 (-) \\ \hline \multirow{3}{*}{CoditT5} & SQL & Token-Level & 64.3 (0.1) & 64.4 (0.2) & 65.4 (0.5) & 66.6 (0.3) & 74.2 (0.4) & 75.3 (0.1) \\  & SQL & Clause-Level & 67.0 (0.4) & 65.4 (0.5) & 71.3 (0.5) & 70.9 (0.2) & 76.3 (0.0) & 77.2 (0.3) \\  & PyDict & Clause-Level & 67.1 (0.2) & 66.5 (0.4) & 70.6 (0.8) & 70.8 (0.6) & 76.3 (0.3) & 77.0 (0.3) \\ \hline \multirow{3}{*}{CodeT5} & SQL & Token-Level & 66.7 (0.9) & 65.9 (0.5) & 68.2 (0.4) & 69.4 (0.8) & 75.6 (0.4) & 76.5 (0.6) \\  & SQL & Clause-Level & 68.3 (0.3) & 68.2\({}^{+}\)(0.6) & 71.8\({}^{+}\)(0.4) & 72.5\({}^{+}\)(0.2) & 76.7 (0.6) & 77.4 (0.3) \\  & PyDict & Clause-Level & 66.6 (0.8) & 67.1 (0.8) & 72.0\({}^{+}\)(0.3) & 72.4\({}^{+}\)(0.2) & 77.3 (0.6) & 77.8 (0.2) \\ \hline \multirow{3}{*}{CodeT5\({}^{*}\) CodeT5\({}^{*}\) CodeT5} & \multirow{3}{*}{PyDict} & \multirow{3}{*}{Program} & **69.2\({}^{+}\)**(0.4) & **68.4\({}^{+}\)**(0.2) & **72.5\({}^{+}\)**(0.4) & **73.1\({}^{+}\)**(0.2) & 77.3 (0.4) & 77.6 (0.6) \\  & & & 69.0\({}^{+}\)(0.2) & 68.2\({}^{+}\)(0.1) & **72.5\({}^{+}\)**(0.3) & 73.0\({}^{+}\)**(0.6) & **78.0\({}^{+}\)**(0.3) & **78.5\({}^{+}\)**(0.3) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Exact Set Match (EM) and Execution Match (EX) accuracy on Spider development set. The **best performances** are in bold and the second bests are underlined. Results with \({}^{+}\) are statistically significant (McNemar’s; \(p<0.05\)) compared to CodeT5-SQL+Token-Level (Appendix D). Otherwise, the results are not statistically significant. \({}^{*}\)We fine-tune the model to generate edit programs only (without resulting queries) and use Python interpreter to execute the edit actions.

 

database grounding is also a major category of errors in our synthesized training set, so our model has learned to resolve similar errors. Nevertheless, it still cannot correct the remaining 38 SQL parses. We notice that our current representation for database schema is missing critical information, such as column data types and foreign key relations, for our error correction model to fix database grounding errors. Following our PyDict representation for SQL, we suggest designing a code representation for database schema that includes such information to tackle this issue in future work.

Structural errors are hard to edit automatically.Besides database grounding, 26 of SmBoP's errors belong to another category, incorrect structure. These 26 samples contain 7 parses with incorrect SQL clauses and 19 parses with incorrect subqueries, but our CodeT5-PyDict+Program model only resolves 1 and 2 of them, respectively. We find that correcting such errors usually involves multiple edit steps, which motivates us to incorporate our model into an interactive framework in future work. As our experiments with simulated user interaction (Appendix E.2) show, when our model interacts with the simulated user to correct one clause at a time, it is able to fully correct more SQL parses. Thus, we deem interactive correction would maximize our model's utility in practice.

## 5 Related Work

Since the release of CodeBERT Feng et al. (2020), many language models of code have emerged for program understanding and generation Ahmad et al. (2021); Chen et al. (2021); Guo et al. (2021); Wang et al. (2021); Guo et al. (2022); Fried et al. (2023); Nijkamp et al. (2023). In addition to program-related tasks, recent work shows they also excel at processing natural language structures. Using code as meaning representations (MRs), we can leverage language models of code in various tasks, such as commonsense reasoning Madaan et al. (2022), action planning Singh et al. (2022), and event extraction Wang et al. (2022). In fact, how to design MRs to reduce model learning difficulty is a salient research question in semantic parsing Guo et al. (2019); Gan et al. (2021b); Nie et al. (2022).

Our work demonstrates that program-related tasks themselves can also benefit from code-based MRs. Specifically, we apply such MRs to SQL error correction, a variant of automatic program repair tasks Tufano et al. (2019); Panthaplackel et al. (2022); Zhang et al. (2023). Although SQL is a code-based MR, it is much harder for models to learn compared to other MRs, such as FunQL and lambda calculus Li et al. (2022). Consequently, without many SQL queries in their pre-training corpora, language models of code can underperform state-of-the-art text-to-SQL parsers. By converting SQL queries into Python dictionaries, we can explicitly represent their compositional substructures and define edit actions as programs, which reduces the learning difficulty for language models of code and yields better performance.

## 6 Conclusion and Future Work

This paper presents a study on developing a text-to-SQL error correction model with clause-level edits and different representations. Our comprehensive experiments demonstrate that _clauses are better semantic units than tokens_ for editing SQL queries and _mimicking patterns in code pre-training corpora_ helps better leverage language models of code. As a future direction, we plan to incorporate our model into interactive semantic parsing frameworks Li et al. (2020); Yao et al. (2019); Yao et al. (2020); Zeng et al. (2020) by suggesting possible edits to users once a wrong parse is identified. In this way, users would more efficiently correct parse errors and get better assistance. We also plan to experiment with other language models of code Fried et al. (2023); Nijkamp et al. (2023) and text-to-SQL datasets Zelle and Mooney (1996); Gan et al. (2021a) to verify the generalizability of our method.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Error Category**} & \multicolumn{4}{c}{**CodeT5**} & \multicolumn{4}{c}{**BRIDGEv2**} & \multicolumn{4}{c}{**SmBoP**} \\ \cline{2-10}  & Resolved & Unresolved & All & Resolved & Unresolved & All & Resolved & Unresolved & All \\ \hline Database Grounding & 15 & 51 & 66 & 14 & 48 & 62 & 16 & 38 & 54 \\ Incorrect Structure & 2 & 15 & 17 & 2 & 12 & 14 & 3 & 23 & 26 \\ Syntax \& Grammar & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 4 & 5 \\ False Negative & 0 & 9 & 9 & 0 & 6 & 6 & 0 & 8 & 8 \\ Other & 1 & 7 & 8 & 2 & 16 & 18 & 1 & 6 & 7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Analysis of 100 sample errors made by each text-to-SQL parser. We group the errors into 5 categories and examine if our CodeT5-PyDict+Program model resolves them.

 



### Limitations

Actual applications of our model.Our work assumes that input SQL queries to our model are always wrong. This assumption is more feasible in an interactive semantic parsing framework, where the users are expected to decide whether a SQL parse, accompanied by its natural language explanations (Elgohary et al., 2020, 2021; Narechania et al., 2021; Mo et al., 2022), has errors or not. Alternatively, to remove this assumption, it would be interesting for future work to study the performance of our error correction model in combination with an automatic error detection model (Chen et al., 2023).

Experiments with more language models of code.We have only experimented with two language models of code, CoditT5 and CodeT5, both using T5-base (Raffel et al., 2020) as their underlying model architecture. It would be interesting to test how our conclusions generalize to other language models of code in the future. Based on the strong capabilities of large language models of code, such as Codecx (Chen et al., 2021), InCoder (Fried et al., 2023), and CodeGen (Nijkamp et al., 2023), we believe that these models can better exploit their knowledge about data structures and their operations in Python. These models may perform even better on Text-to-SQL error correction with our proposed representations.

## Acknowledgements

We would like to thank the anonymous reviewers and colleagues from the OSU NLP group for their thoughtful comments. This research was supported in part by a sponsored award from Cisco Research, NSF IIS-1815674, NSF CAREER #1942980, NSF OAC-2112606, and Ohio Supercomputer Center (Center, 1987). The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein. Ziru is also supported by The Ohio State University Graduate School through University Fellowship.

## References

* Ahmad et al. [2021] Wasi Ahmad, Saikat Chakraborty, Baishakhii Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 2655-2668. External Links: Link Cited by: SS1.
* D. S. Center [1987]O. Supercomputer center. Cited by: SS1.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. Stuch, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. Hebgen, G. Alschol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, V. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. 2021. Evaluating large language models trained on code. Cited by: SS1.
* X. Deng, A. Hassan Awadallah, C. Meek, O. Polozov, H. Sun, and M. Richardson (2021)Structure-grounded pretraining for text-to-SQL. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 1337-1350. External Links: Link Cited by: SS1.
* A. Elgohary, C. Meek, M. Richardson, A. Fourney, G. Ramos, and A. Hassan Awadallah (2021)NL-EDIT: correcting semantic parse errors through natural language interaction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 5599-5610. External Links: Link Cited by: SS1.
* Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou (2020)CodeBERT: a pre-trained model for programming and natural languages. In Findings of the Association

for Computational Linguistics: EMNLP 2020_, pages 1536-1547, Online. Association for Computational Linguistics.
* Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In _The Eleventh International Conference on Learning Representations_.
* Gan et al. (2021a) Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. 2021a. Towards robustness of text-to-SQL models against synonym substitution. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2505-2515, Online. Association for Computational Linguistics.
* Gan et al. (2021b) Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021b. Natural SQL: Making SQL easier to infer from natural language specifications. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2030-2042, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcodet: Unified crossmodal pre-training for code representation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7212-7225, Dublin, Ireland. Association for Computational Linguistics.
* Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Syvatkovskiy, Shengyou Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Dexin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training code representations with data flow. In _International Conference on Learning Representations_.
* Guo et al. (2019) Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
* Husain et al. (2020) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020. Code-searchnet challenge: Evaluating the state of semantic code search.
* Jorgensen and Sheppard (2007) Magne Jorgensen and Martin Sheppard. 2007. A systematic review of software development cost estimation studies. _IEEE Transactions on Software Engineering_, 33(1):33-53.
* Li et al. (2020) Yuntao Li, Bei Chen, Qian Liu, Yan Gao, Jian-Guang Lou, Yan Zhang, and Dongmei Zhang. 2020. "what do you mean by that?" a parser-independent interactive approach for enhancing text-to-SQL. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6913-6922, Online. Association for Computational Linguistics.
* Li et al. (2022) Zhenwen Li, Jiaqi Guo, Qian Liu, Jian-Guang Lou, and Tao Xie. 2022. Exploring the secrets behind the learning difficulty of meaning representations for semantic parsing. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3616-3625, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Lin et al. (2020) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4870-4888, Online. Association for Computational Linguistics.
* Maadau et al. (2022) Aman Macahan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 1384-1403, Abu Dhabil, United Arab Eminrates. Association for Computational Linguistics.
* McNemar (1947) Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. In _Psychometrika_, volume 12, page 153-157.
* Mo et al. (2022) Lingbo Mo, Ashley Lewis, Huan Sun, and Michael White. 2022. Towards transparent interactive semantic parsing via step-by-step correction. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 322-342, Dublin, Ireland. Association for Computational Linguistics.
* Mosbach et al. (2021) Marius Mosbach, Maksym Andriushnchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines. In _International Conference on Learning Representations_.
* Narechania et al. (2021) Arpit Narechania, Adam Fourney, Bongshin Lee, and Gonzalo Ramos. 2021. Diy: Assessing the correctness of natural language to sql systems. In _26th International Conference on Intelligent User Interfaces_, IUI '21, page 597-607, New York, NY, USA. Association for Computing Machinery.
* Nie et al. (2022) Lunyiu Nie, Shulin Cao, Jiaxin Shi, Jiuding Sun, Qi Tian, Lei Hou, Juanzi Li, and Jidong Zhai. 2022. GraphQ IR: Unifying the semantic parsing of graph query languages with one intermediate representation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5848-5865, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics 

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In _The Eleventh International Conference on Learning Representations_.
* Panthaplackel et al. (2022) Sheena Panthaplackel, Milos Gligoric, Junyi Jessy Li, and Raymond Mooney. 2022. Using developer discussions to guide fixing bugs in software. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 2292-2301, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Leer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Rubin and Berant (2021) Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive bottom-up semantic parsing. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 311-324, Online. Association for Computational Linguistics.
* Scholak et al. (2021) Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9895-9901, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In _Proceedings of the 35th International Conference on Machine Learning_, volume 80 of _Proceedings of Machine Learning Research_, pages 4596-4604. PMLR.
* Singh et al. (2022) Ishika Singh, Valts Bluksis, Arsalan Mousavian, Ankitt Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animeesh Garg. 2022. Progr prompt: Generating situated robot task plans using large language models. In _Workshop on Language and Robotics at CoRL 2022_.
* Volume 13_, EMNLP '00, page 133-141, USA. Association for Computational Linguistics.
* Tufano et al. (2019) Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk. 2019. On learning meaningful code changes via neural machine translation. In _Proceedings of the 41st International Conference on Software Engineering_, ICSE '19, page 25-36. IEEE Press.
* Wang et al. (2020) Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7567-7578, Online. Association for Computational Linguistics.
* Wang et al. (2022) Xingyao Wang, Sha Li, and Heng Ji. 2022. Code4struct: Code generation for few-shot structured prediction from natural language.
* Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pretrained encoder-decoder models for code understanding and generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Weiss et al. (2007) Cathrin Weiss, Rahul Premraj, Thomas Zimmermann, and Andreas Zeller. 2007. How long will it take to fix this bug? In _Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007)_, pages 1-1.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Guugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG: 

Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 602-631. External Links: Link Cited by: SS4.
* Z. Yao, Y. Su, H. Sun, and W. Yih (2019)Model-based interactive semantic parsing: a unified framework and a text-to-SQL case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 5447-5458. External Links: Link Cited by: SS4.
* Z. Yao, Y. Tang, W. Yih, H. Sun, and Yu Su (2020)An imitation game for learning semantic parsers from user interaction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 6883-6902. External Links: Link Cited by: SS4.
* P. Yin and G. Neubig (2017)A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 440-450. External Links: Link Cited by: SS4.
* T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev (2018)Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 3911-3921. External Links: Link Cited by: SS4.
* Volume 2, AAAI'96, pp. 1050-1055. External Links: Link Cited by: SS4.
* J. Zeng, X. Victoria Lin, F. C.H. Hoi, R. Socher, C. Xiong, M. Lyu, and I. King (2020)Photon: A robust cross-domain text-to-SQL system. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online, pp. 204-214. External Links: Link Cited by: SS4.
* J. Zhang, S. Panthaplackel, P. Nie, J. Ljasi, and M. Gligoric (2023)Coditt5: pretraining for source code and natural language editing. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE '22, New York, NY, USA, pp. . External Links: ISBN 978-1-467-12-1367-1 Cited by: SS4.

## Appendix A SQL PyDict Representation

We implement the transformation from any SQL query to our PyDict representation in three steps (Section 2.1). First, we use context-free grammar to parse a SQL query and obtain its abstract syntax tree (AST). The AST naturally contains a SQL decomposition where each clause has its unique subtree. In addition, if a clause contains a nested query, it would be represented as another independent subtree, which is a child of the root node in the clause's AST subtree. With these substructures explicitly represented, we use depth-first search to traverse through the AST to build our PyDict representation bottom-up. In other words, if a clause contains a subquery, we process the subquery tree as an independent SQL AST and build a dictionary for it. Then, we combine it with other substructures of the clause with different dictionary keys. For example, in Table F.1, we first build the dictionary for "subquery0" and assign this identifier as the key. In the main "clause," we replace the subquery's corresponding span with this identifier. Finally, we use another dictionary to wrap the main "clause" and "subquery0" together as the final representation of the "where" clause. We repeat this procedure for each clause to incrementally add (key, value) pairs to the dictionary and "store" it to the variable sql, which we refer to in program edit representations.

## Appendix B Text-to-SQL Parser Selection

We choose existing text-to-SQL parsers in our experiments according to two principles: the parsers predict database entity values, and they cover different decoding strategies, including grammar-based (BRIDGEv2), bottom-up (SmBop), and token-based (CodeT5). We did not include parsers using top-down decoders because they usually cannot predict entity values in conditional statements, such as RAT-SQL [20]. Instead, we include BRIDGEv2 because its decoding method mimics 

the left-to-right CFG derivation of a program, and it uses SQL syntax-based constraints to prevent grammatical errors. In recent work, such decoders, also used in PICARD (Scholak et al., 2021), are more popular than top-down decoders.

## Appendix C Implementation Details

Our models (Section 3.2) are implemented in PyTorch (Paszke et al., 2019) using Huggingface (Wolf et al., 2020) and trained on a single NVIDIA RTX A6000 GPU (48GB). We use Adafactor (Shazeer and Stern, 2018) to train all our models with the same hyperparameters adapted from Mosbach et al. (2021):

* Learning rate: \(3e-5\)
* Batch size: 16
* Epochs: 10
* Scheduler: Linear decay with \(10\%\) warmup

## Appendix D Statistical Significance Test

To demonstrate the effectiveness of our three clause-level edit representations (Section 4.1), we perform McNemar's Test (McNemar, 1947) to measure the statistical significance of their results in comparison to CodeT5-SQL+Token-Level. For each significance test between two models, we use the median results among our three runs to calculate the comparison matrix. Then, we compute the \(p\)-values using statsmodels.4 When \(p<0.05\), we reject the null hypothesis. In other words, we consider the accuracy improvement statistically significant when \(p<0.05\).

Footnote 4: [https://www.statsmodels.org/dev/generated/statsmodels.stats.contingency_tables.mcmear.html](https://www.statsmodels.org/dev/generated/statsmodels.stats.contingency_tables.mcmear.html)

## Appendix E Additional Results

Results on our development set.We report model performances on our held-out development set (Section 3.1) in Table E.1. During training, we select the best model by evaluating its EX and EM accuracy on the development set (Section 3.3) every 500 steps. Surprisingly, we find that CodeT5-SQL+Clause-Level sometimes achieves the best performance. For BRIDGEv2, it obtains 35.9 EM accuracy and 39.3 EX accuracy, while CodeT5-PyDict+Program only obtains 34.5 EM accuracy and 37.1 EX accuracy. A possible explanation is that in comparison to the test set, our development set has SQL structures and databases that are more similar to the training set, while the test set has unseen SQL structures and less similar databases. It may also indicate that CodeT5-SQL+Clause-Level overfits the synthetic training data and fails to generalize to realistic test data.

Results for simulated interaction experiments.To show the potential of using our model in an interactive framework, we extend our main experiments (Section 4.1) by adding simulated user interactions. Since our model uses beam search to decode the edit actions \({\bf e}=\left\{e_{1},e_{2},...,e_{n}\right\}\) and the resulting correct SQL query \({\bf q}_{+}\) (Equation 1), we simulate user interactions to select one edit action \(e_{i}\) at a time from the beam results.

At each time step \(t\), we prompt the decoder with previously selected edit actions \(e_{1},...,e_{t-1}\) to complete the sequence \(e_{t},...,e_{n},{\bf q}_{+}\) using beam search with size 3. Then, we use gold SQL annotations to simulate the user interaction, which selects an edit action \(e_{t}\) from the three candidates at step \(t\) or chooses to skip the current step when all three candidates are wrong. If skipping, the user continues to check the consequent edit actions \(e_{t+j}\) (\(j=1,2,...,n-t\)) until it selects the next edit action. When the interaction finishes, we append the selected edit action to the prompt and let the model regenerate a completion with the new prompt for the next step's interaction. Having simulated interactions for all edit actions, we do not use the generated \({\bf q}_{+}\) directly because some edit actions are skipped. Instead, we execute the selected ones on the initial SQL query to derive the final query.

As shown in Table E.2, when collaborating with a simulated user, our error correction model can further improve the base parsers' accuracy. Compared to its performance without using any interactions, our model achieves up to 4.1 point more absolute improvement on EM accuracy (\(72.5\to 76.6\); BRIDGEv2) and 5.0 point more absolute improvement on EX accuracy (\(73.1\to 78.1\); BRIDGEv2). With these results for simulated interaction experiments, we deem that incorporating our error correction model into an interactive framework is a promising future direction.

 



## Appendix F More Representation Examples

We provide two more examples in Table 1 and 2 to demonstrate how we represent SQL with subqueries and their edits (Section 2.2). We also show different representations for _Insert_ and _Delete_ edit actions.





\begin{table} \end{tabular}
\end{table}
Table F.1: Example representations for a wrong SQL query that contains a nested subquery and its edit actions (including Insert edits). The corresponding natural language utterance is “What is the number of cars with a greater accelerate than the one with the most horsepower?”

\begin{table} \end{table}
Table F.2: Example representations for a wrong SQL query and its edit actions (including Delete edits). The corresponding natural language utterance is “Find the name of the employee who got the highest one time bonus."



**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.



C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Appendix B_ C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? \(4\) C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _Appendix B_ D **Did you use human annotators (e.g., crowdworkers) or research with human participants?**:
* _Left blank._
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _No response._
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _No response._
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _No response._
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _No response._
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _No response._