Venue: NAACL 2025

# Distill-C: Enhanced NL2SQL via Distilled Customization with LLMs

 Cong Duy Vu Hoang\({}^{1*}\), Gioacchino Tangari\({}^{2*}\), Clemence Lanfranchi\({}^{3**}\),

Dalu Guo\({}^{2}\), Paul Cayet\({}^{3}\), Steve Siu\({}^{2}\), Don Dharmasiri\({}^{2}\), Yuan-Fang Li\({}^{2}\),

Long Duong\({}^{2}\), Damien Hilloulin\({}^{3}\), Rhicheek Patra\({}^{3}\), Sungpack Hong\({}^{3}\), Hassan Chafi\({}^{3}\)

###### Abstract

The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finentuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks,1Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost.

Footnote 1: Datasets are available at [https://github.com/ClérenceLanfranchi/Distill-C](https://github.com/ClérenceLanfranchi/Distill-C)

0
Footnote 0: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track)_, pages 833–848


* **Targeted Distillation**: Utilizes an ensemble of LLMs to balance their strengths and weaknesses, generating tailored datasets with features like date-time handling, financial analytics, and SQL compliance.
* **Modular Synthesis**: Separates natural language and SQL synthesis, leveraging multiple LLMs for better data diversity and robustness.
* **Quality Assurance**: Uses a multi-step filtering process (pattern matching, execution checks, LLM juries) to refine data quality.

Our Distill-C framework effectively enables small LLMs to perform on par with, or even surpass, their teacher models, exhibiting gains of 36% on average across different families of models and on various challenging benchmarks.

## 2 Methodology

### Customization Scenarios

We present three distinct scenarios, including **AddRef**, **LearnPrior**, and **FixIt** - each of which is based on a reasonable assumption often confirmed in enterprise settings, where product and engineering teams typically have the capacity to provide a few examples, instructional guidance, and error feedback from early model deployments.

**AddRef: Incorporating Reference Examples.** Reference examples consist of a pre-defined subset of natural language (NL) queries provided by the **Customer** and serve as a basis for guiding data generation by LLMs. It is essential that these generated NL examples not only closely resemble the reference examples but also exceed them in complexity and originality.

**LearnPrior: Leveraging Prior Custom Instructions.** The **Customer** provides a limited set of statements detailing prior requirements and expectations for SQL responses generated by NL2SQL models. These statements convey the **Customer's** insights into how model outputs should align with their specific needs.

**FixIt: Distilling Targeted Knowledge from Error Feedback.** In this scenario, the **Customer** has initial access to a baseline model that is evaluated to identify a set of unacceptable model errors. These errors serve as starting points for bootstrapping targeted improvements, helping the model avoid similar issues in subsequent iterations.

### The Distill-C Framework

We developed our **Distilled** Customization framework, abbreviated as **Distill-C**, to synthesize tailored knowledge specifically adapted to the customer scenarios described above. The core components of our proposed **Distill-C** framework are illustrated in Figure 1. The framework comprises distinct NL and SQL synthesizers, followed by a three-stage filtering pipeline, and it enables the integration of knowledge from multiple advanced LLMs at each stage.

#### 2.2.1 Distillation Pipelines

Our framework decouples NL and SQL synthesis, which, though less resource-efficient than

Figure 1: **The Proposed Distill-C Framework.**



a single-step approach, offers two key benefits: _First_, independent generation by different LLMs enhances data diversity; _Second_, it leverages model-specific strengths. For example, while Llama3.1-70B-Instruct excels at generating realistic queries for a database schema, it may miss OracleSQL-specific nuances better addressed by Mixtral-8x22B-Instruct-v0.1, as shown in Table 1.

NL Synthesizer Pipeline.The NL synthesizer produces new NL queries or questions, guided by the customer's customization scenarios, including reference NL examples2 (AddRef); prior expert instructions (LearPrior); and targeted knowledge from error feedback (FixIt). These scenarios can be applied individually or in combination.

Footnote 2: consisting of 100 examples or fewer to initiate the data synthesis process.

The NL synthesis process3 begins with **Reference NL Extraction & Sampler**, where NL queries are sampled from reference examples, balancing inspiration with diversity within the LLM's context window. The **Prompt Constructor** then assembles NL generator prompts by combining these sampled NL examples and a database (DB) schema.4 We also utilize discarded examples from previous generation rounds, incorporating a limited selection of them into the prompt as negative examples, which helps to iteratively refine the natural language synthesis process.

Footnote 3: As further illustrated in Appendix Figure 5.

Footnote 4: sampled from a pool of training DB schemas.

Finally, multiple LLMs (preferably 50B+ parameters) generate diverse NL queries by leveraging high-temperature sampling and varied random seeds, benefiting from their superior instruction following and generation diversity.5 The outcome of the NL synthesis phase is a set of new NL queries relevant to the customer use case, each mapped to a DB schema.

SQL Synthesizer Pipeline.Starting with a set of {NL question, DB schema} pairs generated in earlier steps, the SQL synthesizer employs multiple Generator LLMs to translate each question into its corresponding SQL query. This process produces a preliminary, or "raw", distillation dataset (prior to filtering), where each entry forms a complete NL2SQL data point pairing the DB schema and NL question as the prompt with the SQL query as the completion. This dataset serves as a foundation for transferring knowledge from strong foundational LLMs into smaller models.

Footnote 5: Despite their capability, proprietary LLMs (OpenAI; Anthropic; Gemini) are excluded from this process due to licensing restrictions on production use of their generated data.

Key aspects of the SQL synthesis process (detailed in Appendix Figure 6) include:

* **Diverse LLMs as Generators**: Multiple LLMs enhance data diversity and address model-specific gaps, with some excelling in constructs like the Oracle SQL dialect.
* **Instruction-Conditioned Generation**: Task-specific instructions (LearPrior) ensure SQL outputs align with customer requirements, including handling complex datetime structures (intervals, absolute and relative references).6

Footnote 6: as further illustrated in Appendix Figure 8.

The synthesis process includes three key steps:

1. [leftmargin=*]
2. **Prompt Constructor**: Combines user queries, database schemas, and task-specific instructions to create effective prompts.
3. **SQL Generation**: LLMs generate SQL queries with descriptions, forming a synthetic supervised fine-tuning (SFT) dataset that clarifies complex SQL elements.
4. **Prompt Post-Processing**: Strips instructions from prompts in the SFT dataset to ensure

Figure 2: The Multi-Step Filtering Pipeline in our **Distill-C** Framework.

 

smaller models learn directly from distilled examples.

#### 2.2.2 Multi-Step Filtering Pipeline

The training examples derived from the NL & SQL Synthesizer pipelines, consisting of (i) a prompt with a new question and (ii) an SQL completion, undergo a multi-step filtering process, as illustrated in Figure 2, to ensure data quality and minimize noise:

* **Pattern-Based Filtering**: Removes examples with non-target syntax (e.g., MySQL-specific keywords for Oracle SQL), reducing the load on resource-intensive downstream filters.
* **Execution-Based Filtering**: Validates SQL by executing it on real databases linked to schema contexts, discarding non-executable queries to prevent negatively impacting model performance.
* **LLM-Based Quality Evaluation**: Uses multiple strong LLMs as "juries" [14] to evaluate and rank examples for semantic accuracy to ensure alignment with intended NL meaning. This automated approach replaces manual review for large datasets.
* **LLM-Based Relevance Evaluation**: Ensures examples are relevant to the target use case by requiring unanimous agreement among LLMs. Irrelevant data is flagged as "Filtering Feedback" (Figure 1) for refining the NL synthesis.

#### 2.2.3 Finetuning

The final step involves finetuning the smaller target LLM using synthesized instruction data and a small bootstrapping dataset, which is crucial for mitigating biases and preventing model collapse [6].

## 3 Experiments

### Evaluation Tasks

We evaluate our approach on customer-identified tasks, including:

* **DateTime**: Generating SQL for complex temporal conditions, including relative (e.g., _"last 2 quarters"_) and composite clauses (e.g., _"first quarter of the last 5 years"_).
* **Financial Analytics**: Querying trends, correlations, and financial metric breakdowns (e.g., profits by country or quarter).
* **OracleSQL Compliance**: Producing syntactically correct OracleSQL queries.

### Data and Evaluation Settings

Experimental Data.We built our experimental data using Spider (1.0) [8] and BIRD [9]. For each task, we prepared three datasets: (i) a curated test set; (ii) a small development set for customization via AddRef, LearnPrior, and FixIt scenarios; (iii) a training set generated with the **Distill-C** pipeline. The training, testing and dev sets respectively comprise 199, 31, 10 disjoint DB schemas to prevent data leakage. Data statistics are in Table 3.

Metric.We use execution accuracy [7] to evaluate our framework, which compares the execution results of the generated SQL query and the ground-truth on the corresponding database.

### Model Settings

We evaluated our proposed Distill-C framework with a series of settings, progressing from NL-only (B) to complete (A-Full), which enables systematic evaluation of the impact of increasing supervision

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Model Variant**} & \multicolumn{4}{c}{**DateTime (\%)**} & \multicolumn{4}{c}{**Financial Analytics (\%)**} & \multicolumn{2}{c}{**OraclesQL**} \\  & **spd-ora** & **spd-lite** & **bd-rite** & **bd-ora** & **spd+bor-** & **spd+bf-life** & **spd-ora** & **bd-ors** \\ \hline \multicolumn{8}{c}{**Student ILMs \& SFT with Distill-C (A-Full Setting)**} \\ \hline CodeVowen1.5-7B-Char & 30.4 & 58.1 & 37.9 & 2.6 & 24.8 & 47.8 & 33.9 & 4.6 \\ +Distill-C (A-Full) & **74.0** & **68.7** & **57.2** & **33.8†** & **89.5** & **84.1†** & **77.6** & **34.8†** \\ \hline Llama-3.1-8B-Instruct & 29.8 & 62.6 & 41.3 & 2.6 & 17.0 & 35.9 & 36.1 & 3.1 \\ +Distill-C (A-Full) & **81.2 

and tailored training signals on model performance, as shown in Table 4. The distillation signals from teacher LLMs are derived in Table 5.

### Public Main Results

The experimental results in Table 1 highlight the effectiveness of our proposed **Distill-C** framework, which integrates three customization scenarios (AddRef, LearnPrior, FixIt) to enhance the performance of various student LLMs across three challenging tasks: DateTime, Financial Analytics, and Oracle SQL Compliance. Our approach achieves significant performance gains across three foundational LLMs: CodeQwen1.5-7B-Chat (26.2%, 55.5%, 36.9%), Llama-3.1-8B-Instruct (25.3%, 54.3%, 36.1%), and Mistral-7B-Instruct-v0.3 (29.2%, 59.7%, 31.4%) for DateTime, Financial Analytics, and OracleSQL Compliance, respectively. These improvements across multiple benchmarks underscore the robustness of our method in enhancing LLM capabilities across diverse tasks.

Furthermore, the distilled models surpass several strong out-of-the-box LLMs, including their teacher models such as Qwen2-72B-Instruct, Llama-3.1-70B-Instruct, and Mistral-8x22b-Instruct-v0.1, which can be attributed to the tailored prompts that are used to guide the data synthesis process, fostering better SQL generation from the teacher models. Our fine-tuned models outperform larger state-of-the-art LLMs (e.g., Mistral-Large-Instruct-2407 and DeepSeek-Coder-V2-Instruct) on multiple benchmarks, showcasing the effectiveness of the Distill-C framework. These findings demonstrate the potential of the Distill-C framework to significantly enhance smaller LLMs, enabling them to handle complex tasks more effectively while providing substantial efficiency benefits for deployment.

### Customer Impact

We demonstrated the business impact of our Distill-C method through enhanced performance gains on internal and customer-specific datasets.7

Footnote 7: Due to proprietary restrictions, we are unable to disclose the specifics of the customer schemas as well as benchmark sets for the NL2SQL tasks.

The performance boost of Distill-C on domain-specific tasks, as shown in Table 2, highlights its capability to address key challenges in customer-specific tasks such as DateTime handling, financial analytics, and SQL compliance, improving average accuracy significantly, by 22.6 absolute points. For DateTime tasks in Customer 1's account management use case, Distill-C achieved near-perfect accuracy (97%), demonstrating its robustness in handling temporal data critical for financial workflows. In Customer 2's financial analytics use case, the model significantly improved performance from

\begin{table}
\begin{tabular}{l l} \hline
* & **Description*
* \\ \hline B & Distill-C w/ Addef (NL); Uses 10 to 100 NL–only examples for da synthesis without SQL supervision. \\ \hline C & Distill-C w/ Addef (NL) + Learrinor: Ads tailored instructions to NL-only examples to guide SQL generation. \\ \hline D & Distilled-C w/ Addef (NL+SQL): adds SQL supervision with paired NL-SQL samples for explicit Nu-to-SQL mappings. \\ \hline E & Distil-C w/ Addef (NL) + Learrinor
* Fail: Estends C with incorrect SQL examples to train error recognition. \\ \hline A-Full & Full Distill-C: AddRef (NL+SQL) + Learrinor + Fail \\ \hline \end{tabular}
\end{table}
Table 4: Summary of evaluation settings.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt} c c c} \hline
**Cluster** & **Use Case** & **Student Model** & **Distill-C Model** & **Distill-C Impact** \\ \hline
**Customer 1** & Account payables and receivables management (4 schemes; 192/497 examples with datetime) & 80\% & **97\%** & Distill-C \(\rightarrow\) DateTime \\
**Customer 2** & Information technology services and consulting (1 scheme; 25/28 examples with financial analytics) & 54\% & **78\%** & Distill-C \(\rightarrow\) Financial Analytics \\
**Customer 3** & Autonomous database (6 schemes; 99/99 examples with OracleSQL compliance) & 42\% & **71\%** & Distill-C \(\rightarrow\) OracleSQL Committee \\ \hline  & & & & & \\ \end{tabular}
\end{table}
Table 2: Impact of Our Distill-C Method on Customer Benchmarks.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} c c c} \hline
**Task** & **Origin** & **SQL Dialect** & **Train** & **Dev** & **Test** \\ \hline
**DateTime** & Bird & OracleSQL & 9.621 & 115 & 533 \\  & Bird & SQLite & 33,173 & 78 & 234 \\  & Spider & OracleSQL & 13,460 & 131 & 680 \\  & Spider & SQlite & 37,172 & 97 & 179 \\ \hline
**Financial Analytics** & Bird & OracleSQL & 13,460 & 63 & 1,753 \\  & Bird & SQLite & 23,091 & 113 & 734 \\  & Spider & OracleSQL & 17,734 & 108 & 3,820 \\  & Spider & SQLite & 35,749 & 123 & 1,366 \\ \hline
**OracleSQL** & Bird & OracleSQL & 29,877 & 319 & 1,469 \\
**Compliance** & Spider & OracleSQL & 39,369 & 326 & 1,478 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics of Train, Dev, and Test Datasets.

 

54% to 78%, showcasing its ability to handle complex financial datasets and provide actionable insights. Finally, for Customer 3, focused on OracleSQL compliance in autonomous database use case, Distill-C delivered a substantial gain, raising accuracy from 42% to 71%. These results underscore Distill-C's versatility and effectiveness in enhancing precision and reliability across specialized tasks in diverse domains.

### Ablation Study

We conduct two ablation studies to assess the impact of individual scenarios in Distill-C.

**Individual FixIt Scenario.** We evaluate the FixIt scenario using Llama-3.1-8B-Instruct and CodeQwen1.5-7B-Chat as student LLMs. Errors identified from the Spider training set Yu et al. (2018) are processed through our data generation pipeline, where the NL Prompt Constructor (Figure 5) utilize these errors to guide teacher LLMs (Qwen2-72B-Instruct for CodeQwen and Llama-3.1-70B-Instruct for Llama) to create targeted datasets used to finetune the student models, producing FixIt-enhanced versions.

On the Spider development set, FixIt achieves performance improvements of 6.4% and 8% for Llama-3.1-8B-Instruct and CodeQwen1.5-7B-Chat, respectively, significantly narrowing gaps with their teacher models. Figure 3 shows notable gains in Analytics and Set Operations, effectively addressing key weaknesses.

**Full Scenarios.** Figure 4 demonstrates the consistent and substantial improvements achieved by integrating all scenarios AdRef+LearnPrior+FixIt) within our Distill-C framework. While the AddRef scenario alone (Setting B) already brings a significant improvement of 24.7% on average, showcasing the importance of finentuning models on tasks that are similar to the target tasks, we also see that providing prior knowledge and leveraging errors is key to obtaining optimal performance. Moreover, the similarity in performance between scenarios C and D (respectively +30.4% and +32.6% on average) tends to show that custom instructions and examples of ground truth SQL queries are both valid options to distill prior knowledge. This integration leads to significant performance gains across a diverse range of benchmarks, including DateTime, Financial Analytics, and Oracle SQL Compliance, showcasing the versatility and robustness of our approach. Notably, these improvements are observed consistently across multiple student LLMs, underscoring the generalizability and effectiveness of the proposed framework. Overall, the results highlight how the synergistic combination of these scenarios enables Distill-C to address complex challenges and deliver superior outcomes, making it a compelling solution for advancing language understanding and task-specific performance.

## 4 Related Work

Recent advancements in NL2SQL research have explored techniques to enhance the performance of

Figure 3: **FixIt Ablation Study Experiments. Performance of student models finetuned with the FixIt scenario using Distill-C on Spider (dev) sub-groups, showing results for student, finetuned, and teacher models, with sample counts per group.**



Large Language Models (LLMs).

**Prompt Engineering and Reasoning.** Prompt engineering has been explored to optimize NL2SQL capabilities of LLMs. PET-SQL [14] adopts a two-round framework with enhanced representations, and EPI-SQL [35] generates error-prevention prompts to reduce LLM errors. Self-correction and iterative refinement have also been explored in SQL-CRAFT [60] and DART-SQL [79], which integrate interactive feedback loops. However, these approaches are not well-suited to smaller Large Language Models (LLMs) because they necessitate acute reasoning capabilities that such models typically lack. On the other hand, Distill-C addresses this limitation by focusing on bridging the performance gap between large and small LLMs. This method leverages the advanced reasoning abilities of larger LLMs to distill their knowledge into more compact forms, thereby enhancing the capabilities of smaller models without requiring extensive computational resources.

**Synthetic Data Generation.** Recent works have shown the great promise of synthetic data. SQL-GEN [84] produces dialect-specific synthetic training data, while SENSE [36] utilizes synthetic data for domain generalization and preference learning. Our approach focuses on creating tailored datasets that cater to specific customer needs by integrating targeted instructions and relevant examples into our data generation pipeline. Unlike previous work, we further customize the data generation process for individual student language models (LLMs) using error-driven reference examples.

## 5 Conclusion

We introduce Distill-C, a novel customizable distillation framework for enhancing small LLMs in NL2SQL tasks for enterprise applications. Despite their smaller sizes, the enhanced models by Distill-C achieve significant gains over strong baselines across benchmarks, including DateTime, Financial Analytics, and Oracle SQL Compliance. The initial costs associated with Distill-C, which involve hosting larger LLMs for data generation and fine-tuning smaller models, are offset by long-term advantages. These benefits arise because business units can then utilize more efficient and specialized smaller LLMs, ultimately leading to a substantial return on investment. Our work lays the foundation for robust distillation solutions, enabling the development of specialized NL2SQL models that can be tailored to specific business needs.

Our future work will explore extensions to preference alignment training [17] and applications to other practical tasks.

#### Acknowledgments

We extend our sincere appreciation to our colleagues at the Science Team within Oracle Cloud Infrastructure (OCI) for their support and valuable feedback.

Figure 4: **Ablation study with distillation settings (Table 4). Notations: spd: Spider, bd: Bird, dt: DateTime, ana: Analytics, ora: OracleSQL, like: SQLite, cpl: Compliance. Numerical results are reported in Appendix B.3.

 

We are grateful to Giulia Carocari for her assistance in translating SQL queries between SQLite and Oracle SQL, as well as the anonymous reviewers whose valuable feedback significantly improved this work.

## References

* Chen et al. (2023) Hailin Chen, Amrita Saha, Steven Hoi, and Shafiq Joty. 2023. Personalized distillation: Empowering open-sourced LLMs with adaptive learning for code generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6737-6749, Singapore. Association for Computational Linguistics.
* Gerstgrasser et al. (2024) Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. 2024. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. _Preprint_, arXiv:2404.01413.
* Li et al. (2024) Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiyang Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2024a. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. In _Proceedings of the 37th International Conference on Neural Information Processing Systems_, NIPS '23, Red Hook, NY, USA. Curran Associates Inc.
* Li et al. (2024b) Zhishuai Li, Xiang Wang, Jingjing Zhao, Sun Yang, Guoqing Du, Xiaoru Hu, Bin Zhang, Yuxiao Ye, Ziyue Li, Rui Zhao, and Hangyu Mao. 2024b. Pet-sql: A prompt-enhanced two-round refinement of text-to-sql with cross-consistency. _Preprint_, arXiv:2403.09732.
* Liu et al. (2024) Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. 2024. A survey of nl2sql with large language models: Where are we, and where are we going? _Preprint_, arXiv:2408.05109.
* Liu and Tan (2024) Xiping Liu and Zhao Tan. 2024. Epi-sql: Enhancing text-to-sql translation with error-prevention instructions. _Preprint_, arXiv:2404.14453.
* Mao (2024) Toby Mao. 2024. Sqlglot: Python sql parser and transpiler. [https://github.com/tobymao/solglot](https://github.com/tobymao/solglot). Accessed: 2024-11-29.
* Mao et al. (2024) Wenxin Mao, Ruiqi Wang, Jiyu Guo,ichuan Zeng, Cuiyun Gao, Peiyi Han, and Chuanyi Liu. 2024. Enhancing text-to-SQL parsing through question rewriting and execution-guided refinement. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 2009-2024, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Minaee et al. (2024) Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. _Preprint_, arXiv:2402.06196.
* Powerza et al. (2024) Mohammadreza Powerza, Ruoxi Sun, Hailong Li, Lesly Miculicich, Tomas Pfister, and Sercan O. Arik. 2024. Sol-gen: Bridging the dialect gap for text-to-sql via synthetic data and model merging. _Preprint_, arXiv:2408.12733.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: your language model is secretly a reward model. NIPS '23, Red Hook, NY, USA. Curran Associates Inc.
* Verga et al. (2024) Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. _Preprint_, arXiv:2404.18796.
* Xia et al. (2024) Hanchen Xia, Feng Jiang, Naihao Deng, Cunxiang Wang, Guojiang Zhao, Rada Mihalcea, and Yue Zhang. 2024. \({}^{\textit{rd}}\): "this is my sql, are you with me?" a consensus-based multi-agent system for text-to-sql tasks. _Preprint_, arXiv:2402.14851.
* Yang et al. (2024a) Jiaxi Yang, Bingyuan Hui, Min Yang, Jian Yang, Junyang Lin, and Chang Zhou. 2024a. Synthesizing text-to-SQL data from weak and strong LLMs. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7864-7875, Bangkok, Thailand. Association for Computational Linguistics.
* Yang et al. (2024b) Jiaxi Yang, Bingyuan Hui, Min Yang, Jian Yang, Junyang Lin, and Chang Zhou. 2024b. Synthesizing text-to-sql data from weak and strong llms. _Preprint_, arXiv:2408.03256.
* Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. _arXiv preprint arXiv:1809.08887_.
* Zhong et al. (2020) Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-SQL with distilled test suites. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 396-411, Online. Association for Computational Linguistics.
* Zhu et al. (2024) Xiaohu Zhu, Qian Li, Lizhen Cui, and Yongkang Liu. 2024. Large language model enhanced text-to-sql generation: A survey. _Preprint_, arXiv:2410.06011.

 

Additional Figures

We include additional figures to illustrate the components of our Distill-C framework: the NL Synthesizer Pipeline in Figure 5 and the SQL Synthesizer Pipeline in Figure 6, respectively.

## Appendix B Additional Tables

### Experimental Setup: Training and Inference Configurations

We also provide our training and inference hyperparameter configurations in Table 8.

### Evaluation Tasks

In Table 6, we present detailed descriptions and examples of the three evaluation tasks used to assess the impact of our **Distill-C** framework.

### Ablation Study Evaluation

We further provide the detailed results of our ablation study (shown in Figure 4) in Table 7.

## Appendix C SQL Dialect Conversion

We utilize the SQLGlot library Mao et al. (2024) to translate SQL queries from the Bird and Spider datasets from SQLite to the OracleSQL dialect. To enhance the translations, we apply a custom postprocessor to address potential parsing issues and align with OracleSQL conventions.

## Appendix D Prompts in The Distill-C Framework

### Prompts for NL and SQL Synthesizer Pipelines

We also present additional prompt templates utilized across various components of our Distill-C framework, including:

* An example prompt template for the NL Synthesizer pipeline AddRef scenario).
* An example prompt template for the SQL Synthesizer pipeline LearnPrior scenario) with a focus on DateTime use case.

### Prompts for Multi-Step Filtering Pineline

Given the large scale of the Synthetic SFT Data (over 10,000 instances), manual or human-in-the-loop evaluation is not feasible. Therefore, we rely on soft evaluation using multiple strong LLMs as judges, following Verga et al. (2024). We employed two primary evaluation phases as shown in Figure 2 as follows:

* **LLM-based Quality Evaluation.** In this evaluation, each 'judge' LLM assigns a 1-to-5 star score per criterion, with a cut-off as a hyperparameter: consensus on '5 stars' is required for SQL correctness and compliance, and at least '4 stars' for NL quality (Figure 9).
* **LLM-based Relevance Evaluation** This evaluation step queries multiple LLMs to assess the relevance of a generated example to the use case in the Reference Examples, using prompts in Figure 10. Examples marked 'relevant' by all LLMs are added to the final synthetic fine-tuning set, while those marked 'irrelevant' are stored as 'irrelevant examples' for the Input Schema to guide future NL generation (Figure 5).

 



\begin{table}
\begin{tabular}{l l c p{142.3pt}} \hline
**OracleSQL** & Handling OracleSQL-dialect syntax, including ORDER BY with “FETCH FIRST/LAST \(\left\{\text{N}\right\}\) & car_1 (Spider) & What are the different models created by either General Motors or over \\  & & & & SELECT DSTINTECH T1. ”model” FROM model_list T1 JOIN car_makers T2 ON \\  & & & General Motors or over \\  & ROWS*, correct quoting, and casing for schema object names. & 3500 los? & T1\^{}_model= T3. ”model” JION cars_data \\  & & & T4 ON M3."Maskedized = T4."id” WHERE T2.FullName = “General Hotors’ OR \\  & & & T4Weight > 3500 \\ \cline{2-6}  & financial (Bird) & List out the accounts who have the earliest trading date in 1995 ’ & SELECT account_id FROM turns WEREXTRACT(YEAR FROM “Gate”) = 1995 ORDER \\ \end{tabular}
\end{table}
Table 6: Details of the Evaluation Tasks.



Figure 5: The NL Synthesizer Pipeline in our **Distill-C** Framework.

Figure 6: The SQL Synthesizer Pipeline in our **Distill-C** Framework.

 



\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Finetuning Configuration**} \\ \hline
**Pretrained Checkpoints** & CodeQwen1.5-7B-Chat, Llama3.1-8B-Instruct, Mistral-7B-Instruct-v0.3 \\
**Batch Size** & 512 examples per step & & & \\
**Learning Rate** & 1e-6 (with linear decay) & & & \\
**Warmup Steps** & 2,000 & & & \\
**Max Sequence Length** & 8192 tokens & & & \\
**Optimizer** & Paged AdamW 8-bit (\(\beta_{1}=0.9\), \(\beta_{2}=0.95\)) & & N/A & & \\
**Gradient Clipping** & 1.0 & & & & \\
**Training Steps** & 20,000 & & & & \\
**Evaluation Metrics** & Checkpoint-based Execution Accuracy & & & & \\
**Hardware Setup** & 8 NVIDIA A100 40GB GPUs & & & \\ \hline \multicolumn{4}{c}{**Inference Configuration**} \\ \hline
**Decoding Strategy** & Random Sampling & & & \\
**Temperature** & 0.5 & & & \\
**Top-k Sampling** & 40 & & & \\
**Top-p Sampling** & 0.9 & & & \\
**Max Sequence Length** & 2048 tokens & & & \\
**Batch Size** & 32 & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 7: **Performance comparison of model variants on DateTime, Financial Analytics, and OracleSQL tasks for the different distillation scenarios. Notations: OOTB (Out-Of-The-Box), spd (Spider), bd (Bird), ora (OracleSQL), lite (SQLite).**

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Model Variant**} & \multicolumn{4}{c}{**DateTime**} & \multicolumn{4}{c}{**Financial Analytics**} & \multicolumn{2}{c}{**Oracles SQL**} \\  & **spd-ora** & **spd-lite** & **bd-lite** & **bd-ora** & **spd+bd-ora** & **spd+bld-lite** & **spd-ora** & **bd-ora** \\ \hline  & \multicolumn{4}{c}{**Qwen1.5-7B-Chat**} \\ \hline OOTB-Baseline & 30.4 & 58.1 & 37.9 & 2.6 & 24.8 & 47.8 & 33.9 & 4.6 \\ SFTed-Setting-B & 49.2 & 60.3 & 45.3 & 14.5 & 73.4 & 77.5 & 62.7 & 18.9 \\ SFTed-Setting-C & 67.4 & 62.6 & 46.6 & 24.4 & 78.7 & 77.8 & 70.1 & 28.0 \\ SFTed-Setting-D & 65.7 & 61.5 & 42.7 & 26.5 & 85.9 & 78.1 & 69.8 & 31.2 \\ SFTed-Setting-E & 72.9 & 60.5 & 44.5 & 29.1 & 85.4 & 79.1 & 66.8 & **37.0** \\
**SFTed-Setting-A-Full** & **74.0** & **68.7** & **57.2** & **33.8** & **89.5** & **84.1** & **77.6** & 34.8 \\ \hline  & \multicolumn{4}{c}{**Llama3.1-8B-Instruct**} \\ \hline OOTB-Baseline & 29.8 & 62.6 & 41.3 & 2.6 & 17.0 & 35.9 & 36.1 & 3.1 \\ SFTed-Setting-B & 44.2 & 66.5 & 45.7 & 19.2 & 74.6 & 75.9 & 57.4 & 18.0 \\ SFTed-Setting-C & 61.3 & 63.1 & 47.8 & 33.3 & 76.9 & 76.7 & 72.2 & 30.8 \\ SFTed-Setting-D & 69.1 & 69.3 & 48.9 & **35.5** & 78.1 & 74.1 & 72.9 & 35.8 \\ SFTed-Setting-E & 68.5 & 62.6 & 50.2 & 34.6 & 80.8 & 75.6 & 73.3 & **40.8** \\
**SFTed-Setting-A-Full** & **81.2** & **67.6** & **59.3** & 29.5 & **83.2** & **78.2** & **79.4** & 32.0 \\ \hline  & \multicolumn{4}{c}{**Mistral-7b-v0.3-Instruct**} \\ \hline OOTB-Baseline & 22.1 & 46.4 & 22.2 & 2.6 & 21.1 & 24.5 & 38.4 & 4.4 \\ SFTed-Setting-B & 66.3 & 63.1 & 36.5 & 23.9 & 82.6 & 70.7 & 68.4 & 28.8 \\ SFTed-Setting-C & 69.6 & 60.9 & 37.4 & 31.2 & 81.3 & 76.6 & 71.2 & 33.7 \\ SFTed-Setting-D & 74.0 & **70.9** & 37.4 & 35.9 & 82.0 & 79.7 & 71.3 & 35.1 \\ SFTed-Setting-E & **81.8** & 64.5 & **39.0** & **40.6** & 62.0 & 23.8 & 73.7 & **38.5** \\
**SFTed-Setting-A-Full** & 74.6 & 65.4 & 38.8 & 31.2 & **84.5** & **80.4** & **77.3** & 28.2 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Configuration details for training and inference in our experiments.



* [258] M. C. Bender, J. R. Friedman and A. S. G. Wilson, _The Theory of Computing_, volume 130 of _Lecture Notes in Computer Science_, vol. 497 of _Springer Series in Mathematics_, pages 6-100. Springer Berlin Heidelberg, 2000.
* [259] D. E. Brown and P. W. Hastings, _A Course on the Structure of a Lattice_, volume 130 of _Lecture Notes in Computer Science_, vol.

 

Here is the database schema context:

CREATE TABLE ports(  id INT,  name VARCHAR(255),  country VARCHAR(255)  ) CREATE TABLE cargoes(  id INT,  name VARCHAR(255),  tontage INT, port_id INT,  load_date DATE  ) DateTime Instructions:  - With a DATE_column, refer to the following instructions:  - 'today': TRUNC(DATE_column) = TRUNC(SYSDATE)  - 'yesterday': TRUNC(DATE_column) = TRUNC(SYSDATE)-1  - 'tomorrow': TRUNC(DATE_column) = TRUNC(SYSDATE)+1  - 'this year': EXTract(YEA FROM DATE_column) = EXTRACT(YEAR FROM SYSDATE)  - 'this month': TO_CHAR(DATE_column, 'YYYY-MM') = T0_CHAR(SYSDATE, 'YYYY-MM')  - 'last month': TO_CHAR(DATE_column, 'YYYY-MM') = TO_CHAR(ADD_MONTHS(SYSDATE, -1)  - 'veryT-MM')  - 'next month': TO_CHAR(DATE_column, 'YYYY-MM') = TO_CHAR(ADD_MONTHS(SYSDATE, +1)  - 'VVYY-MM')  - 'until last month' TO_CHAR(DATE_column, 'YYYY-MM') <= TO_CHAR(ADD_MONTHS(SYSDATE, -1)  - 'VVYY-MM')  - 'until next month' TO_CHAR(DATE_column, 'YYYY-MM') <= TO_CHAR(ADD_MONTHS(SYSDATE, +1)  - 'VYY-MM')  - 'this quarter': TRUNC(DATE_column, 'Q') = TRIUNC(SYSDATE, 'Q')  - 'last quarter': TRUNC(DATE_column, 'Q') = ADD_MONTHS(TRUE(SYSDATE, 'Q'), -3)  - 'last year': TO_CHAR(DATE_column, 'YYYY') = TO_CHAR(ADD_MONTHS(SYSDATE, -12), 'YYYY')  - 'in the last X months': DATE_column BETWEEN ADD_MONTHS(SYSDATE, -X) AND SYSDATE  - 'in the last X quarters': DATE_column ADD_MONTHS(SYSDATE, -\({}^{\prime}\)), -3*x) AND SYSDATE  - 'in next X days': [TRANUC(DATE_column) BETWEEN TRUNC(SYSDATE) AND TRUE(SYSDATE) + X]  - 'in year XXX': EXTRACT(YEA FROM DATE_column) > XXXX  - 'after year XXX': EXTRACT(YEAR FROM DATE_column) > XXX  - 'day X of month Y of year': To_CHAR(DATE_column), 'YYYY-MM-DD') = 'ZZZZ-YM-MX'  - 'after day X of month Y of year': DATE_column > TO_DATE( 'ZZZZ-YY-MX', 'YYYY-MM-DD')  - 'next week': TO_CHAR(dueDate, YYYY-IN") = TO_CHAR(SYSDATE + 7, 'YYYY-IN')  - 'in this February: EXTRACT(Year FROM DATE_column) = EXTRACT(YEAR FROM SYSDATE) AND  - 'in this October: EXTRACT(YEAR FROM DATE_column) = EXTRACT(YEAR FROM SYSDATE) AND  EXTRACT(ONTH FROM DATE_column) = 10  - 'in last February': EXTRACT(YEAR FROM DATE_column) = EXTRACT(YEAR FROM SYSDATE) - 1 AND  - 'in next February': EXTRACT(YEAR FROM DATE_column) = EXTRACT(YEAR FROM SYSDATE) + 1 AND  - 'next charscetnam': EXTRACT(WARTRODATE_column) = 2  - 'from this April': TRUNC(DATE_column, 'MM') >= ADD_MONTHS(TRUNC(SYSDATE, 'YYYY'), 4-1) # beginning of this year + 3 months to align with start of April (EXTRACT(MONTH not needed here)  - 'from this January': TRUNC(DATE_column, 'MM') >= TRIUNC(SYSDATE, 'YYYY') # beginning of this year + 0 months to align with start of January (EXTRACT(MONTH not needed here)  - 'from this October': TRUNC(DATE_column, 'MM') >= ADD_MONTHS(TRUNC(SYSDATE, 'YYYY'),  10-1) # Beginning of this year + 9 months to align with start of October  (EXTRACT(MONTH not needed here)  - 'untiltilis February': TRUNC(DATE_column, 'MM') <= ADD_MONTHS(TRUNC(SYSDATE, 'YYYY'),  2-1) # beginning of this year + 1 months to align with start of February  (EXTRACT(MONTH not needed here)  ... (truncated) Question: What's the total tonnage of all cargoees loaded or unloaded at the port of Singapore before last April Write a SQL query in Oracle SQL dialect, compatible with the latest version of Oracle Database, that answers the question above.

 

Given an input Question and a Oracle SQL query, prepare an assessment based on the following criteria:

SQL Correctness  - Add one star if the Oracle SQL query returns incorrect results  - Add one more star, i.e. award 2 stars if the Oracle SQL query executes but returns   partially correct results  - Add one more star, i.e. award 2 stars if the Oracle SQL query returns mostly correct  results but with minor inaccuracies or omissions  - Add one more star, i.e. award 2 stars if the Oracle SQL query returns correct results with negligible issues  - Add one more star, i.e. award 2 stars if the Oracle SQL query returns accurate and complete results as per the requirement

Compliance with Oracle SQL Standards  - Add one star if the SQL query does not follow Oracle SQL standards or best practices,  using deprecated or non-standard syntax  - Add one more star, i.e. award 2 stars if the SQL query loosely follows Oracle SQL  standards, with several deviations from best practices.  - Add one more star, i.e. award 2 stars if the SQL query generally follows Oracle SQL  standards but has room for better alignment with best practices.  - Add one more star, i.e. award 2 stars if the SQL query closely follows Oracle SQL  standards and adhere to many best practices.  - Add one more star, i.e. award 2 stars if the SQL query strictly adheres to Oracle SQL  standards and best practices, showcasing exemplary coding standards.

Quality of the Natural Language Query  - Add one star if the natural language query does not match the SQL, or cannot be  answered given the provided Schema.  - Add one more star, i.e. award 2 stars if the natural language query matches the SQL,  but the question does not make any sense to be asked (totally unrealistic).  - Add one more star, i.e. award 3 stars if the natural language query is consistent with  the SQL, but it it does not look natural (no domain knowledge, the style looks  synthetic-templated, does not use "domain-specific" words).  - Add one more star, i.e. award 4 stars if the natural language query is correct and  consistent, but the NL Question can further be improved for clarity, conciseness,  small typos.  - Add one more star, i.e. award % stars if the natural language query is perfect.

The Schema context is provided below. CREATE TABLE ports(  id INT,  name VARCHAR(255),  country ARCHAR(255) ) CREATE TABLE cargoes(  id INT,  name VARCHAR(255),  tonange INT,  port_id INT,  load_date DATE  ) Question: What's the total tonage of all cargoees loaded or unloaded at the port of Singapore  before last April  Oracle SQL: SELECT SUM(c.tonnage) FROM cargoes c JOIN ports p ON c.port_id = p.id WHERE  p.name = 'Singapore' AND (EXTRACT(YEAR FROM c.load_date) < EXTRACT(YEAR FROM SYSDATE) -  1 OR (EXTRACT(YEAR FROM c.load_date) EXTRACT(YEAR FROM SYSDATE) - 1 AND EXTRACT(MONTH  FROM c.load_date) < 4)); The output must have following items in an orderly manner:  - The final star ratings of criterions in a list-wise manner  - The final star ratings of criterions in a json format  - Explain the scores with a short text (< 100 words).

Figure 9: Prompt for LLM-based Quality Evaluation.

 

Given an a Natural Language query and the corresponding SQL Query generated for a NL2SQL Model, your goal is to assess whether the generated example is relevant to the Customer use-case represented by any of the Reference Examples shown below.

Reference Examples - show the distance of the flights that arrived before last May - visits made past more than twelve days - show a list containing staff names and their respective genders who were assigned 2 days ago - Find the names of the university which has more faculties in 2002 than every university in Orange county. - What is all the information about employees hired until June 21, 2002?  - Show me the aircraft names that travelled 8430 kms that departed before November of 4 years ago  - How many students exist who are registered with just a single allergy?  - show all maintenance contracts that end until next Dec  - Give me the list of actors which was last updated until last Saturday  - show the distance of the flights that arrived before last January  - show all machines made in 1992  - Show me invoices that are due to be paid in the next half year.  - what is all the information about employees hired until June 21, 2002?  - show all order items delivered before last march  - Show the number of attendees in year 2008 or 2010.  - Show me all students who registered for a course from 3 days ago, including the course name and student details.  - give people addresses who lived on address till april.  - List all customers who placed an order from the next 30 days and the order status is 'New'.  - show all maintenance contracts that end until next May  - How many customers are not responded to mailshot sent from week 5 2018
Input Natural Language query and SQL query Natural language query: What's the total tonnage of all cargoes loaded or unloaded at the port of Singapore before last April SQL Query: SELECT SUM(c.tonnage) FROM cargoes c JOIN ports p ON c.port_id = p.id WHERE p.name  = 'Singapore' AND (EXTRACT(YEAR FROM c.load_date) < EXTRACT(YEAR FROM SYSDATE) - 1 OR  (EXTRACT(YEAR FROM c.load_date) EXTRACT(YEAR FROM SYSDATE) - 1 AND EXTRACT(MONTH FROM  c.load_date) < 4));
## Assessment ("**Relevant**"/"**Irrelevant**")

Figure 10: Prompt for LLM-based Relevance Evaluation.