Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation (2024)

Abstract: Large Language Models (LLMs) driven by InContext Learning (ICL) have significantly improved the performance of text-to-SQL. Previous methods generally employ a two-stage reasoning framework, namely 1) schema linking and 2) logical synthesis, making the framework not only effective but also interpretable. Despite these advancements, the inherent bad nature of the generalization of LLMs often results in hallucinations, which limits the full potential of LLMs. In this work, we first identify and categorize the common types of hallucinations at each stage in text-to-SQL. We then introduce a novel strategy, Task Alignment (TA), designed to mitigate hallucinations at each stage. TA encourages LLMs to take advantage of experiences from similar tasks rather than starting the tasks from scratch. This can help LLMs reduce the burden of generalization, thereby mitigating hallucinations effectively. We further propose TA-SQL, a text-to-SQL framework based on this strategy. The experimental results and comprehensive analysis demonstrate the effectiveness and robustness of our framework. Specifically, it enhances the performance of the GPT4 baseline by 21.23% relatively on BIRD dev and it yields significant improvements across six models and four mainstream, complex textto-SQL benchmarks. For reproducibility, we release our code and prompt at https:// github.com/quge2023/TA-SQL.

heading: 1 Introduction
text: In the age of big data, relational databases, as carriers for storing massive amounts of data, play a crucial role in information processing and data analysis. Text-to-SQL, which aims to convert natural language (NL) queries to executable SQL queries, facilitates access to ubiquitous relational data for a broader range of non-technical users, thereby attracting remarkable attention (Cai et al., 2018; Yu et al., 2018a; Wang et al., 2020; Cao et al., 2021).
Recently, Large Language Models (LLMs) have
shown impressive success on a wide range of NLP tasks through in-context learning (ICL) (Dong et al., 2022), such as question answering (Nair et al., 2023; Nguyen et al., 2023), logic reasoning (Khot et al., 2023; Zhao et al., 2023), and code generation (Gu, 2023; Chen et al., 2023). The application of LLMs has also improved the performance of text-to-SQL to another level of intelligence (Dong et al., 2023; Pourreza and Rafiei, 2024; Gao et al., 2023). Delving into their crafted designs, these works generally approach text-toSQL through a two-stage paradigm. The first stage, Schema Linking, involves the precise mapping of natural language queries to the relevant entities within a database schema (Lei et al., 2020; Wang et al., 2022; Liu et al., 2021). This meticulous alignment is crucial for the following execution of the query and provides transparency by illustrating how natural language queries are interpreted in relation to the database schema. The second step is Logical Synthesis, which refers to the process of generating accurate SQL queries based on the understanding of the logic of the natural language query and the structure of the database (Yin and Neubig, 2017).
Nevertheless, hallucination, a notorious problem in LLMs that refers to the generation of content that is irrelevant, erroneous, or inconsistent with user intents (Huang et al., 2023), remains a considerable barrier to current frameworks as a reliable automatic text-to-SQL parser. In this work, we first study and conclude primary hallucinations presented in the aforementioned two stages of current text-to-SQL frameworks and attribute them to two main categories: schema-based hallucinations and logic-based hallucinations, as shown in Table 1. Schema-based hallucinations refer to hallucinations in which LLMs might inaccurately identify schema structures, introduce unnecessary attributes, or fail to accurately represent or interpret database values. On the other hand, logic-based
5456
hallucinations can also prevent LLMs from executing accurate JOIN operations, applying appropriate SQL clauses such as GROUP BY and nested sub-queries, or computationally reasoning in data science queries.
The aforementioned challenges reinforce the demand for a robust text-to-SQL framework to minimize hallucinations and improve overall performance while maintaining interpretability. We posit that hallucinations often arise when models misinterpret the decomposed stages of a task as entirely new challenges, for which they lack prior training. This situation is comparable to human experiences, where unfamiliarity with a task can lead to disorientation and a higher propensity for errors (Silva et al., 2016). Thus, just as experienced individuals can
draw on familiar situations to reduce cognitive load and enhance task performance (Carbonell, 1993), we introduce Task Alignment (TA), a novel strategy to mitigate hallucinations of LLMs in this way. TA fundamentally adjusts the approach of models to unfamiliar tasks by aligning them with tasks they have previously trained on. This method reduces the dependence of models on their generalization capability for generating responses from scratch, thereby significantly reducing the incidence of hallucinations.
We further propose a text-to-SQL framework named TA-SQL, which consists of a Task-Aligned Schema Linking (TASL) module and a TaskAligned LOGical synthesis (TALOG) module. TA is employed to mitigate hallucinations in these
two modules, respectively, enhancing the performance of the framework while preserving its interpretability. Experimental results on four textto-SQL datasets and our comprehensive analysis demonstrate the effectiveness and robustness of TA-SQL. Specifically, TA-SQL relatively improves the performance of the GPT4 baseline in terms of Execution Accuracy (EX) by 21.23% and 14.86% on BIRD (Li et al., 2024) and SPIDER (Yu et al., 2018b), respectively. Moreover, our experimental results also illustrate that TA-SQL is a modelagnostic framework, exhibiting applicability to both mainstream closed-source LLMs and opensource weaker LLMs.

heading: 2 Preliminaries
text: Problem Definition Given a natural language question Q = { q1, ..., q|Q| } with its corresponding database schema D = ⟨C, T ⟩, where C ={ c1, ..., c|C| } and T = { t1, ..., t|T | } represent columns and tables, |C| and |T | refer to the number of columns and tables in each database respectively. The goal of text-to-SQL is to generate the corresponding SQL query y.
In-Context Learning In-Context Learning (ICL) is a paradigm that allows language models to learn tasks with only a few examples in the form of demonstrations (Dong et al., 2022), or even without examples. It requires no additional training and
is directly applicable to pre-trained LLMs. In this work, we only discuss hallucinations in ICL-based text-to-SQL frameworks. Few-shot prompting is a scenario in ICL where it uses task descriptions I and a set few-shot input-output (I/O) prompting demonstrations S = {(x1, y1), ..., (xk, yk)} to assist LLM M to solve the input problem x which belongs to a task m by:
y = fM(x, I, S | m), (1) where fM (· | m) refers to a mapping function applied by LLM M when it generalizes task m from scratch. When I/O pairs are no longer provided, the scenario transitions to zero-shot prompting, where the model is expected to understand and complete the task relying solely on its pre-trained knowledge, and the output of zero-shot prompting could be represented as:
y = fM(x, I | m), (2)

heading: 3 Methodology

heading: 3.1 Task Alignment
text: Inspired by the human approach of drawing upon relevant past experience when tracking unfamiliar tasks, we introduce Task Alignment (TA), a novel strategy designed to mitigate hallucinations. The fundamental idea is that LLMs have already acquired knowledge of various tasks during training
(Ouyang et al., 2022). We refer to tasks for which the basic rules have been mastered by LLMs as pre-trained tasks. Given a novel task mn and one of its problems x, TA first retrieves the most related pre-trained task mp from a set of LLM pre-trained tasks {mp1,mp2, ...,mpk}. In this study, we manually select mp for each new task. The potential for LLMs to automatically select tasks is a valuable prospect for future research. It then leverages this pre-trained task to reconstruct the representation for the novel task mn, aligning it with the representation that the LLMs are familiar with. The goal of TA is to solve the problem x by:
y = fM(x, I, S | mn → mp), (3)
where fM (· | mn → mp) refers to the mapping function applied by LLM M when it applies experiences from aligned pretrained task mp while generalizing mn.
TA explicitly guides LLMs to approach unfamiliar tasks from the perspective of more familiar ones, alleviating the burden of from-scratch generalization and subsequently mitigating hallucinations.

heading: 3.2 TA-SQL
text: We further propose a robust text-to-SQL framework named TA-SQL. TA-SQL adheres to the two-stage paradigm of previous work, consisting of a taskaligned schema linking module and a task-aligned logic synthesis module. However, unlike previous works (Dong et al., 2023; Pourreza and Rafiei, 2024; Gao et al., 2023) that treat each decomposed module as an entirely new task for the LLM to generalize from scratch, we apply the TA strategy to each module to stimulate incremental generalization of LLMs. This design not only mitigates hallucinations effectively for better performance but also maintains the interpretability of the entire model. We introduce these two modules, respectively, in the following sections. The prompts employed within each module are displayed in Appendix A.

heading: 3.2.1 Task-Aligned Schema Linking Module (TASL)
text: Given a natural language question Q with corresponding database schema D, schema linking is responsible for identifying references to columns, tables, and condition values in Q. However, LLMs are not adept at the schema linking task. Therefore, when dealing with complex databases characterized by their extensive size and abundant semantic information, LLMs are highly prone to generating
schema-based hallucinations. (Gan et al., 2023). Hallucinations at this stage would be influential negatively on the final performance due to the error propagation (Caselli et al., 2015).
As shown in Figure 1 (b), we design a TASL module for the schema linking stage. The schema linking task in this module is represented as first generating a dummy SQL query and then extracting related schema entities from it as the final output. Although the schema linking task is not familiar to LLMs, its downstream task, SQL generation, has been extensively exposed during training (Guo et al., 2024). Playing a similar role to negative sampling in the skip-gram algorithm (Mikolov et al., 2013), the objective of dummy SQL generation is not to create executable SQL directly for the final application. Instead, its primary function is to subtly leverage the successful experiences of schema entity selection during the generation process for LLMs.

heading: 3.2.2 Task-aligned Logical Synthesis Module (TALOG)
text: The TALOG module is responsible for reasoning the transformation logic from the NL query into SQL based on the results generated by the TASL module and accordingly producing accurate SQL. This process often involves multiple forms of logic, including SQL syntax reasoning, external knowledge reasoning, and computational reasoning. Such complexity presents a significant challenge for LLMs, leading to the emergence of logic-based hallucinations (Lee, 2023).
In fact, SQL serves as a tool for extracting values from Relational Database (RDB) for data analysis. It encapsulates various data analysis logics, such as data filtering, mathematical computation, and output synthesis. As such, we employ the TA in the capacity of a data scientist who addresses complex problems through step-by-step logical operations using pandas-like APIs (Zan et al., 2022) and generates symbolic representations that include reasoning processes, as shown in Figure 1 (c).
After logic alignment with data analysis processes, the remaining challenge is to ensure that LLMs are proficient in perceiving valid SQL syntax and structures. This proficiency is crucial for the generation of accurate SQL. To facilitate this, we replace conventional pandas API functions with symbolic functions that resemble SQL keywords, thereby enabling the symbolic representation to invoke them effectively.

heading: 4 Experiments

heading: 4.1 Experiment Settings
text: Datasets. We evaluate our text-to-SQL framework on four challenging benchmarks for crossdomain SQLs. (1) BIRD (Li et al., 2024) is the most challenging lager-scale cross-domain text-toSQL benchmark. It has two settings, with and without external knowledge, to highlight the new challenges brought by external knowledge. In this paper, we use its development set for evaluation, which contains 1534 pairs of text-to-SQL data and 11 databases, as the test set is not released. (2) SPIDER(Yu et al., 2018b) is a more standard cross-domain text-to-SQL benchmark. It contains 1034 examples, which cover 20 complex databases across multiple domains, in the development set. (3) DK (Gan et al., 2021a) requires text-to-SQL parsers to equip with the capability of domain knowledge reasoning. (4) REALISTIC removes and switches the obvious mentions of schema items in questions, making it closer to the real scenarios.
Metrics. Following the prior study (Yu et al., 2018b; Li et al., 2024), we use Execution Accuracy (EX) to measure the performance of our method. EX can reflect whether a predicted SQL is valid and return the exact result as the execution result of the ground truth SQL.
Models. We experiment our proposed method with both closed-sourced LLMs and open-sourced code generation models. For the closed-source LLMs, we experiment with GPT family models including ChatGPT (gpt-3.5-turbo) (Ouyang et al., 2022), GPT4 (gpt-4-32k) (Achiam et al., 2023), GPT4-Turbo (gpt-4-turbo), and Claude (claude-2.0) (Anthropic, 2023). For open-source weaker LLM models, we experiment with two most popular and strong baselines, CodeLlama (codellama-34b-instruct) (Roziere et al., 2023), and DeepSeek (deepseek-coder-33b-instruct) (Guo et al., 2024).
Compared Methods. We also compare our method with two SOTA ICL-based methods, that are, DIN-SQL (Pourreza and Rafiei, 2024) and DAIL-SQL (Gao et al., 2023) on both BIRD and SPIDER.
Implementation We implement the schema linking module with zero-shot prompts and the logical synthesis module with 6-shot prompts. For
METHOD DEV TEST
w/o knowledge Palm-2 18.77 24.71 Codex 25.42 24.86 ChatGPT 24.05 26.77 ChatGPT+COT 25.88 28.95 Claude-2 28.29 34.60
all models we used in this paper, we set the argument temperature and top-p as 0 and 1, respectively, to promise reproduction. The max_tokens (max_new_tokens) for closed-source LLMs and open-source weaker LLMs are both set as 800, respectively, for all modules.

heading: 4.2 Main Results
text: Results on BIRD. Table 2 displays the performance of TA-SQL and other competitive methods on the current most challenging text-to-SQL benchmark, BIRD. First, in the setting with oracle knowledge, we demonstrate that TA-SQL effectively mitigates hallucinations in the GPT4 baseline, resulting in a relative improvement of 21.23% in EX on the development set and 7.74% on the test set. This demonstrates that even the most powerful LLMs can produce severe hallucinations during the text-to-SQL process, thereby highlighting the value of hallucination mitigation research. Surprisingly, TA-SQL equipped with GPT4 outperforms the SOTA LLM-based method without finetuning by 2.61% even without the application of self-consistency or re-modification mechanisms. Furthermore, even in the setting without external knowledge, TA-SQL achieves performance comparable to the GPT4 baseline equipped with oracle external knowledge. This suggests that addressing hallucinations within the existing knowledge could
be a promising and cost-effective solution, rather than resorting to the addition of manually extracted external knowledge from heterogeneous resources with much more effort.
Results on SPIDER and its Variant Datasets As shown in Table 3, TA-SQL effectively enhances the EX performance of the GPT4 baseline by 14. 86%, 11. 80%, and 8. 31% on the SPIDER and its variant datasets, DK and REALISTIC, respectively, with improvements across all difficulty levels. This suggests that TA-SQL, as a general method, is not only useful in complex text-to-SQL scenarios that closely mirror the real world but also delivers robust performance on standard text-to-SQL benchmarks where the context is relatively simple.
Results on Model Agnosticism. TA-SQL is proved to be model-agnostic since it can work among mainstream closed-source LLM and opensource weaker language models, as shown in Table 4. TA-SQL can improve the performance across queries of varying difficulty levels for closedsource LLMs. However, we observe that the performance gains brought by TA-SQL for weaker models (CodeLlama, DeepSeek) are relatively limited. This can be attributed to their constrained capabilities in generalizing and instruction-following (Qi et al., 2023), which limit the effectiveness of TASQL in diminishing hallucinations for challenging queries (Shen et al., 2024).

heading: 4.3 Imperative of Two-stage Paradigm
text: We conduct an imperative analysis of the two-stage paradigm. Table 5 illustrates that the two-stage paradigm not only makes the text-to-SQL framework interpretable, but also impacts the overall performance of the framework. Specifically:
The schema linking module constitutes a prerequisite for the success of TA-SQL. Firstly, the removal of the schema linking module disrupts the interpretability of the framework, preventing it from correcting hallucinations through more flexible methods such as human-computer interaction. Secondly, through quantitative analysis,
we discover that the removal of the schema linking module leads to a significant performance decline across queries of varying difficulty levels (↓ 6.42% in total). This is attributed to the fact that more accurate schema linking results not only reduce schema-based hallucinations but also facilitate the subsequent logical synthesis module to conduct more granular and complex reasoning based on these results.
The logical synthesis module determines the upper bound for the performance of the entire framework. This is evidenced by the observation that, relative to the performance decline on simple queries (↓ 1.55%), the removal of the logical synthesis module has a more obvious impact on moder-
ate (↓ 10.68%) and challenging queries (↓ 3.74%). Relatively challenging queries often contain more complex analytical intentions, involving mathematical computations, multi-step reasoning, or compositional generalization. The symbolic representation produced by the logical synthesis module effectively guides analytical reasoning processes, thereby raising the upper bound of the framework’s ability to solve complex problems.

heading: 4.4 Ablation Study
text: After validating the imperative of the two-stage paradigm, we further conduct an ablation study to evaluate the effectiveness of implementing the TA strategy within these two stages. The schema linking module and logical synthesis module, following the customized designs in DIN-SQL (Pourreza and Rafiei, 2024) and NatSQL(Gan et al., 2021b), are implemented, respectively, for comparison.
Results in the Schema Linking Stage. We implement the schema linking module of DIN-SQL in both the zero-shot and few-shot settings for comparison. Three metrics are introduced to facilitate a more intuitive comparison of the schema linking results: (1) Recall computes the ratio of instances in which the schema linking outcomes encompass all ground truth schema elements of this instance. (2) Precision quantifies the accuracy of the linked schema. (3) F1 Score represents a harmonic mean of recall and precision. The detailed definitions of these metrics are presented in Appendix B.
Figure 2 illustrates that, even when following the design of the SOTA method DIN-SQL, zeroshot schema linking tasks confuse LLMs, since it requires LLMs to comprehend and generalize this unfamiliar task from scratch. This confusion can be alleviated by human-annotated example demon-
strations in the few-shot setting (F1 Score: 75.34 → 81.08). However, the TASL module in the zeroshot setting can directly achieve results that are competitive with the few-shot schema linking module in DIN-SQL without human intervention. This suggests that TA can effectively guide the model to align with pre-trained knowledge to tackle unfamiliar tasks without relying on additional external information.
Results in the Logical Synthesis Stage. We implement another logical synthesis module with a classic customized symbolic representation designed in NatSQL (Gan et al., 2021b) and refer to it as NatLOG module. Same as TALOG, it is also implemented in the 6-shot setting. As shown in Figure 3, the TALOG module exhibits superior performance across various levels of difficulty compared with the NatLOG module. This is because the custom symbolic representation includes new rules that the model needs to understand and learn from scratch, thereby increasing the emergence of hallucinations during the logical synthesis process.

heading: 4.5 Fine-grained Case Study
text: To illustrate the effectiveness of TA-SQL qualitatively, we conduct an analysis of its performance across databases in the BIRD development set. TA-SQL demonstrates a significant impact on databases debit_card_specializing, toxicology, and california_schools, yielding relative EX increases of 59.05%, 43.1%, and 32.13%, respectively. From the perspective of database schema, as shown in Table 6 Case 1, GPT4 selects satscores.dname and AVG(AvgScrRead) as final attributes of the generated SQL, which contradicts the intent of the NL query. This suggests that GPT4 struggles to map NL query entities to the database schema
within complex contexts, leading to the occurrence of schema contraction and attribute overanalysis, which fall under schema-based hallucinations. However, TA-SQL can correspond the question entities to the correct column names through precise retrieval of tables and columns by the TASL module, coupled with more granular schema-related reasoning, such as element selection, by the TALOG module. From the perspective of query difficulty, the capacity of TA-SQL to mitigate logicbased hallucinations can yield a more substantial effect within databases that contain complex queries, which require multiple logical operations. In Case 2, GPT4 manifested erroneous computational logic, which is an instance of mathematical delusion, suggesting GPT4’s limited capability when confronted with complicated multi-step reasoning. Conversely, TA-SQL clearly demonstrates the data manipulation process, thereby equipping it with the capacity to manage complex logic.

heading: 4.6 Discussion about Hallucinations in Text-to-SQL Systems
text: Given that this is the first attempt to systematically study hallucinations in text-to-SQL systems, we define them following the survey (Ji et al., 2023). All types of hallucinations shown in Table 1 are categorized as Intrinsic Hallucinations, which occur when the generated SQL query contradicts the information or intent expressed in the natural language query, the underlying database schema, or SQL syntax.
It is worth noting that the distinction between hallucinations we define and errors is quite subtle. While hallucinations can lead to the occurrence of errors, they may not always directly result in errors. For instance, joining redundant tables can sometimes produce the same executed results, which are considered correct in the current SQL evaluation system. More importantly, errors are typically detected after the final SQLs are outputted and executed, usually when the entire workflow is completed. However, hallucinations can be observed and mitigated before final result generations (i.e., during schema linking or logical synthesis phases). Our proposed method originates from this problem definition and achieves equivalent or better performance than error-corrected methods, as demonstrated in the experiment section. Figure 4 shows the fine-grained performance of TA-SQL on mitigating hallucinations across each category.

heading: 5 Related Work
text: Text-to-SQL The development of a successful cross-domain text-to-SQL parser fundamentally involves the creation of an encoder for learning representations of questions and schema and a decoder for generating SQL queries (Qin et al., 2022). For instance, RATSQL (Wang et al., 2020), SDSQL (Hui et al., 2021), LGESQL (Cao et al., 2021), S2SQL (Hui et al., 2022), and Proton (Wang et al., 2022) have advanced the representation learning of natural language questions and database schema using a relational graph neural network. The intro-
duction of sequence-to-sequence pre-trained language models (PLMs) such as T5 (Raffel et al., 2020) and NQG-T5 (Shaw et al., 2021) significantly transforms text-to-SQL tasks, given their adaptability and generative capabilities across diverse datasets. These models demonstrate impressive results through fine-tuning with minimal effort. Besides, PICARD (Scholak et al., 2021) designs a constrained decoder to reject inadmissible tokens at the decoding step. RASAT (Qi et al., 2022) further improves the structural information encoding of T5 by integrating schema alignment into the encoder, while Graphix (Li et al., 2023b) has equipped T5 with multi-hop reasoning. RESDSQL (Li et al., 2023a) enhances T5 by decoupling the schema linking and the skeleton parsing.
Recently, large language models (LLMs) (Ouyang et al., 2022; Chowdhery et al., 2023; Anthropic, 2023) have attracted considerable attention due to their robust reasoning and domain generalization capabilities. Models like DIN-SQL (Pourreza and Rafiei, 2024) and DAIL-SQL (Gao et al., 2023) with few-shot demonstrations, along with the evolution of language models to language agents (Deng et al., 2024; Gu et al., 2024), have pioneered text-to-SQL solutions to a new level of intelligence.
Hallucination One of the most prominent challenges is hallucination, a phenomenon where a model generates information that is not present or inferred from the input (Ji et al., 2023). This issue is particularly serious in text generation tasks, as evidenced by the factual consistency problems of dialog generation (Dziri et al., 2021; Rashkin et al., 2021; Shuster et al., 2021) when using LLMs. As one of the important techniques in database applications, hallucination can result in the text-toSQL generation of erroneous or non-sensical SQL queries.

heading: 6 Conclusion
text: In this research, we first systematically identify and classify common hallucination types in text-toSQL. Subsequently, we propose Task Alignment (TA), a novel strategy to mitigate hallucinations in Large Language Models (LLMs) during the textto-SQL process. Based on this strategy, we further propose TA-SQL, a framework to mitigate hallucinations at each stage of this process. Experimental results and comprehensive analysis show the importance of hallucination research in text-to-SQL and data science and suggest promising directions for future work.

heading: 7 Limitations
text: Our findings suggest that TA is particularly adept at handling complex cases where the knowledge evidence supplied by the BIRD database is explicit and the questions poised are unequivocally answerable, as shown in Section 4.5. On the contrary, in the databases codebase_community, student_club, and thrombosis_prediction, we observe that the clarity of questions and the sufficiency of knowledge evidence render it more effective to directly generate SQL queries from annotated data, bypassing the need for a multi-step calibration process. Furthermore, the selection of familiar tasks in each phase of the text-to-SQL conversion process is currently conducted by human prior knowledge rather than an automated mechanism capable of identifying and retrieving relevant and familiar tasks for TA applications. This gap highlights a notable avenue for future research.

heading: 8 Acknowledgement
text: We thank all constructive comments from anonymous reviewers. Reynold Cheng, Ge Qu, Jinyang Li, and Nan Huo were supported by the Hong Kong Jockey Club Charities Trust (Project 260920140), the University of Hong Kong (Project 109000579), and the HKU Outstanding Research Student Supervisor Award 2022-23. Chenhao Ma was partially supported by NSFC under Grant 62302421, Basic and Applied Basic Research Fund in Guangdong Province under Grant 2023A1515011280. Ge Qu and Jinyang Li were supported by HKU Presidential PhD Scholar Programme. Ge Qu was also funded by Hong Kong PhD Fellowship Scheme.

heading: 9 Ethical Statement
text: All datasets employed in this work are publicly accessible, ensuring the transparency and reproducibility of our findings. Furthermore, the output generated by our investigations is structured as SQL queries—a programming language format—rather than natural language text, which could potentially involve harmful or biased content. Our team meticulously examines each output to confirm the absence of politically sensitive or biased material. Finally, regarding our analysis of open-source models utilizing GPUs, such as Deepseek and Codellama, it is notable that our approach involves only model inference without training.

heading: A TA-SQL Recipe
text: A.1 Pythonic Prompt Unlike previous works that used natural language to construct prompts, all prompts used in TASQL are pythonic prompts. Considering that textto-SQL is fundamentally a code generation task, pythonic prompts can exhibit data structures more clearly and express constraints and requirements more concisely.
A.2 TASL Module The schema linking task in TASL module is represented as first generating a dummy SQL query and then extracting related schema entities from it as the final output. As discussed in Section 4.4, we implement the TASL module in the zero-shot setting, leveraging the efficient employment of TA. The zero-shot prompt used to generate dummy SQL in this module is presented in Figure 5.
Specifically, we employ a Python dictionary to represent the database schema, where the key is the table_name.column_name entity (e.g. account.account_id), and the value is the
comprehensive description of the corresponding column. However, directly concatenating all related information, such as column type, original column description, and value description, as the final comprehensive description might lead to an excessively lengthy prompt. This verbosity could potentially confuse LLMs. Therefore, to prevent such issues, as a preparatory step to generating dummy SQL, we first prompt LLMs to generate a succinct description for each column, drawing upon the aforementioned related information. These succinct descriptions then serve as the value for each column within the database schema dictionary. The prompt used to generate succinct column descriptions is presented in Figure 6.
A.3 TALOG Module
TALOG module employs pandas APIs to guide LLMs in conducting step-by-step logical reasoning. However, there is a natural gap between the conventional pandas APIs and the ultimate execution language, SQL. To bridge this, we replace the conventional pandas API functions with symbolic
functions that resemble SQL keywords, thereby ensuring a precise translation from generated symbolic representations to SQLs in subsequent steps. To facilitate the model’s understanding of this substitution, we provide a few demonstrations, specifically six shots, for the model to learn from, thereby generating the desired symbolic representations. Figure 7 presents the prompts used within TALOG for generating symbolic representations.

heading: B Details of Evaluation Metrics for Schema Linking
text: Recall. Recall P is defined as the proportion of queries for which the linked schema outputted by the schema linking module contains all the ground truth schema, relative to the overall number of queries. It is noteworthy that since the schema retrieved by the TASL module would replace the original complete database schema as the input for the subsequent TALOG modules, the recall determines the upper bound of the EX for the final generated SQL. Considering the ground truth schema set as Sn of the nth query, and the linked schema set as Ŝn, EM could be computed by:
R =
∑N n=1 I(Ŝn, Sn)
N (4)
where I(Ŝn, Sn) is an indicator function, which can be represented as
I(Ŝ, S) = { 1, Ŝ ⊇ S 0, Ŝ ⊉ S
(5)
Precision. Precision P quantifies the accuracy of the linked schema. Considering the ground truth schema set as Sn with length Ln of the nth query, and the linked schema set as Ŝn with length L̂n, precision is computed by:
P = ∑N n=1 pn N , pn = ∑L̂n j=1 I(ŝj ∈ Sn) L̂n (6)
F1 score. F1 score F1 represents a harmonic mean of recall and precision. It offers an evaluation of schema linking results, taking into account both precision and recall as:
F1 = 2 · P ·R P +R
(7)
C Implement Details
The open-source models are implemented using Pytorch1, Transformers2, and vllm3. To
1https://pytorch.org/ 2https://huggingface.co/docs/transformers/en/installation 3https://github.com/vllm-project/vllm
# Understand the pandas-like IR first. Then convert the IR in to executable SQL based on the question, the schema and the imported keywords. # Notice the order of the action in IR may not same as the executable SQL. Make sure the generated SQL is executable and can answer the question accurately according to the schema. # Only select the thing that the question required. Do not select any non-requested stuff. # You may need to look back to the column_description and schema to get the correct value used
in the final SQL
from CLAUSE_KEYWORDS import select, from, where, group by, order by, union, limit, having, distinct, as, between, like, all, on, partition by from JOIN_KEYWORDS import inner join, left join from WHERE_OPERATIONS import is, not, null, none, in, =, >, <, >=, <=, !=, <> from DATE_OPERATIONS import now, curdate, strftime from UNIT_OPERATIONS import -, +, *, / from COND_OPERATIONS import and, or, case, iif from SQL_OPERATIONS import avg, count, max, min, round, sum, abs, length, cast, substr, cast, instr from ORDER_OPERATIONS import desc, asc
column_description = {column_description}
foreign_keys = {foreign_key_dic}
question = {question} schema = {schema} SR = "{SR}" # IR ignore 'join' action. Automatically add a join action if needed to ensure that the generated SQL
executes correctly and has correct syntax. SQL = "SELECT
expedite the inference process, we also implemented deepspeed4. The DeepSeek and CodeLlama models are accessed via huggingface5.
4https://github.com/microsoft/DeepSpeed 5https://huggingface.co/models
5471

heading: Future Work
text: This gap highlights a notable avenue for future research