SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging

## Abstract

Abstract Recent advances in Text-to-SQL have largely focused on the SQLite dialect, neglecting the diverse landscape of SQL dialects like BigQuery and PostgreSQL. This limitation is due to the diversity in SQL syntaxes and functions, along with the high cost of collecting and curating SQL-specific training data. To address this, we introduce SQL-GEN, a framework for generating high-quality synthetic training data for any SQL dialect, guided by readily available dialect-specific tutorials. SQL-GEN significantly improves cross-dialect Text-to-SQL performance, boosting execution accuracy by up to 20% over existing methods. This performance gain narrows the gap with models trained on large-scale human-annotated data. Furthermore, combining synthetic data from SQL-GEN with human-annotated data yields additional improvements of up to 5.6%. To unify multi-dialect capabilities within a single model, we propose a novel Mixture-of-Experts (MoE) initialization that leverages the shared knowledge across dialects. Our approach merges self-attention layers from dialect-specific models and initializes expert gates using dialect-specific keywords. This leads to a versatile model optimized for multiple SQL dialects, outperforming single-dialect models and significantly enhancing overall performance.

## 1 Introduction

The ability to seamlessly interact with databases using natural language is a long-sought goal in the field of human-computer interaction. Text-to-SQL systems, which translate natural language questions into executable SQL queries, are crucial for achieving this goal, bridging the gap between human communication and the structured language of databases (Androutsopoulos et al., ; Hristidis et al., ; Li & Jagadish, ). These systems have become essential components of modern conversational agents, empowering them to efficiently process complex queries within large-scale databases (Yu et al., ; Gu et al., ; Pérez-Mercado et al., ). Moreover, they serve as invaluable copilots for data science professionals, boosting productivity and enabling non-technical users to glean insights without SQL expertise (Li et al., ; Sun et al., , ; Wang et al., ).

SQL has been adopted by each database product (e.g. PostgreSQL, MySQL, and SQLite) to suit their specific needs. Despite their common foundations, these SQL dialects differ significantly in their syntax, functions, and capabilities, which even make the automated translation of queries across dialects a complex task that often requires human intervention (Zmigrod et al., ; Ngom & Kraska, ). Figure 1 exemplifies a question that can be answered with different SQL keywords across different dialects with their own unique keywords that are distinct from one another. Additionally in Section A.7, we provide some of the dialect specific keywords for BigQuery, PostgreSQL, and SQLite, which are not supported across all of them. In the realm of Text-to-SQL, most benchmarks are based on the SQLite dialect, chosen for its simplicity and self-contained nature (Li et al., ; Yu et al., ; Zhong et al., ; Chang et al., ; Gan et al., ).
This dialect dependency poses a significant challenge, as models trained on SQLite-specific syntax are prone to generating erroneous queries in other dialects. A conventional solution involves translating queries across dialects before training, using tools like SQLglot parser or tools offered by cloud providers (Li et al., ; Mao, ; Zmigrod et al., ). However, most of these tools are not 100% successful in translating the queries between dialects Zmigrod et al. (). For example, during the translation of the BIRD benchmark (Li et al., ) from SQLite to BigQuery, approximately 20% of the queries encountered errors using the SQLGlot parser. Additionally, this approach fails to leverage the unique capabilities of each SQL dialect, as queries originally written for SQLite may not fully exploit the potential of the target dialects due to the absence of support for specific functions and keywords in the source dialect. For example, REGEX operations are supported in BigQuery but not in SQLite, so we cannot get this REGEX support by translating queries from SQLite.

Figure: Figure 1: Exemplification of a question being answered using different SQL keywords for different dialects, BigQuery, PostgreSQL,and SQLite.
Refer to caption: extracted/5896809/figures/dialect_specific_queries.png

To address the aforementioned challenges, we propose SQL-GEN, a novel framework for dialect-agnostic synthetic generation of Text-to-SQL pairs for any database.
SQL-GEN consists of a three major steps. Initially, we begin with a small collection of seed SQL templates. These retain only the SQL keywords, abstracting away the specific database schema and values. Accompanying them, we provide dialect-specific tutorials on each SQL keyword across different dialects.
In the first stage, we leverage a Large Language Model (LLM) to expand the seed templates, using the tutorials to adapt the keywords and function to various SQL dialects.
In the second stage, these database-independent templates are populated with actual values and schema elements from any given database.
The final stage involves rigorous quality checks to ensure that the generated pairs accurately match each other.
We conduct thorough evaluations to assess SQL-GEN’s effectiveness in teaching models new dialects, with a particular focus on dialect-specific features such as keywords. We assess the quality of synthetic samples by comparing them against both prior synthetic and human-annotated data. For three dialects, we evaluate various LLMs (of sizes 7B-22B), trained on these pairs as well as other baselines. We show that LLMs trained on SQL-GEN’s synthetic data exhibit a performance increase ranging from 4% to 27%, surpassing those trained on others. For under-explored dialects, we focus on evaluations on real-world data, specifically designed for them. We demonstrate that models trained SQL-GEN’s synthetic data consistently outperform others by a significant margin, approximately 7.5% on BigQuery and 2.5% on PostgreSQL dialect-specific datasets. This highlights the generalizability of SQL-GEN to unseen datasets due to its broad coverage. We also explore data augmentation for cross-domain Text-to-SQL as another use case of synthetic queries.
By integrating synthetic queries with samples from other databases, we show improvements in models’ ability to adapt across domains.
We show consistent performance improvements, up to $\sim$5.7%, fine-tuning with the augmented data.

Each SQL dialect has distinct keywords and functions, rendering a model trained on a specific dialect uniquely specialized. A significant challenge arises when users manage databases across multiple dialects, as deploying multiple dialect-specific models can be computationally demanding. Moreover, we multi-dialect datasets might share common SQL characteristics, leading to some overlap in the features learned by the models. By merging models, we believe they can gain a deeper understanding of core features as they appear across multiple dialects. To overcome this, we introduce a novel method for utilizing the Mixture of Experts (MoE) architecture (Fedus et al., ; Riquelme et al., ; Jiang et al., ).
Specifically, our approach is based on initializing the MoE model using the layers of the dialect-expert models, while the sub-layers are initialized using a two-by-two Spherical Linear Interpolation (SLERP) of self-attentions from the dialect experts, as approaches for efficient merging. Additionally, to harness dialect-specific expertise effectively, we initialize the routers with hidden states corresponding to dialect-specific keywords.
We demonstrate an improvement of 2.5% in average performance compared to other model merging approaches, as well as superior performance, outperforming the expert models up to 7.3%.

## 2 Methodology

We first introduce the SQL-GEN pipeline, designed to generate high-quality, dialect-specific Text-to-SQL samples, as illustrated in Figure 2 and detailed in Algorithm 1. The generation of multi-dialect Text-to-SQL synthetic data addresses the critical issue of insufficient high-quality data for training models tailored to specific SQL dialects. However, since users of Text-to-SQL systems often work with databases across various dialects, serving models in a multi-dialect environment presents a unique challenge. Additionally, many SQL dialects adhere to standard SQL and share common syntaxes, making the concept of information sharing particularly compelling. To this end, we propose a model merging method that combines dialect-specific models into a single, unified MoE model capable of serving multiple SQL dialects effectively.

Figure: Algorithm 1 SQL-GEN: Dialect-Specific Synthetic Question-SQL Pair Generation.

Figure: Figure 2: SQL-GEN to generate diverse and high-quality synthetic Text-to-SQL samples for any database.
Refer to caption: extracted/5896809/figures/data_generation_pipeline.png

### 2.1 Synthetic Text-to-SQL Data Generation

The initial step of SQL-GEN involves creating a pool of simple queries by extracting template question-SQL pairs. Building on this, we expand the templates using LLMs and dialect-specific tutorials, rather than relying solely on extracted templates. After expanding these templates, each one is converted into an actual SQL query, and a corresponding question is generated using an LLM with the sample database. Subsequently, all generated question-SQL pairs, along with their execution results, undergo a quality-checking step to ensure they accurately match each other and effectively extract valuable information from the database. Throughout this process, we apply filtering to remove low-quality samples at each step, to ensure the overall generated data would be high quality.

##### Extraction of Seed Templates:

Similar to (Wu et al., ; Yu et al., ), we extract SQL templates by abstracting all of the schema mentions in the queries from the Spider dataset to serve as a foundational pool for generating more diverse queries.
As the seed queries are initially in SQLite, for the other dialects, we transpile these queries using the SQLGlot parser (Mao, ) before extracting their SQL query templates.

##### Templates Expansion From Tutorials:

The initial pool of query templates created in the previous step presents two main challenges. First, the extracted templates are derived from simple SQL queries, which are relatively basic compared to the queries found in other SQLite benchmarks like BIRD (Li et al., ). Second, for dialects other than SQLite, the seed
templates—originally designed for SQLite would not have complete coverage for all the dialect-specific SQL functions from other dialects. To address these, we expand the templates for each dialect using LLMs with in-context learning (Brown et al., ; Wei et al., ). To prepare the LLMs for template expansion, we first scrape online tutorials for each target dialect, focusing on the use of dialect-specific SQL functions and keywords. We then randomly select a seed template from the pool, pairing it with a random tutorial document about a dialect-specific keyword or function, and prompt the LLM to increase the complexity of the template, drawing inspirations from the document. To ensure the validity of the templates for all of the different dialects, we parse all generated SQL templates using dialect-specific parser (from SQLglot (Mao, )). Figure 3 exemplifies this template expansion step for the BigQuery dialect. Additionally, Section A.9.1 provides the prompt that has been used for template expansion.

Figure: Figure 3: An example of template expansion using BigQuery tutorials and seed templates.
Refer to caption: extracted/5896809/figures/template_expansion.png

##### Sample Generation:

After generating the SQL templates, our next step is to convert them into valid question-SQL pairs. For this process, we select a template along with a database schema with a random row from any given database. Random database rows are necessary since the LLM should be able to fill conditions with actual database values. The database schemas can be sourced from different datasets (e.g. publicly-available Spider or BIRD). As these are originally in SQLite, these databases are migrated to each target dialect for dialects other than SQLite. The combination is then passed to an LLM, instructing it to integrate schema mentions into the templates and generate corresponding questions that align with the SQL queries.
After generating the SQL queries, heuristic-based semantic and syntactic filters are applied to ensure the high quality of both the queries and questions. The specifics of these filters are detailed in the Section A.6. Additionally, Section A.9.2 includes the detailed prompt which is used in this step.

##### Data Quality Checking:

To ensure high quality generation of question-SQL pairs, we present the pairs alongside the first $K$ rows of their execution results over the database to an LLM. This LLM is tasked with verifying that the pair match appropriately and that the question is free of ambiguity. To avoid repeating the same errors, we employ a different LLM, not used in previous steps, to act as the judge. Section A.4.1 provides a detailed analysis of the importance of utilizing a secondary LLM and highlights the importance of this step. Section A.9.3 provides the prompt that has been used for quality checking.

### 2.2 Dialect Experts Merging: Multi-dialects Mixture of Expert (MoE)

With SQL-GEN, we can generate question-SQL pairs for various dialects and train corresponding dialect-specific models. However, in real-world scenarios, users often manage databases across different dialects, necessitating the deployment of multiple models, which can come with practical challenges, including increased model serving costs and overhead of managing multiple checkpoints. Additionally, while each dialect features unique keywords and functions, there is commonality across some SQL keywords across dialects that can be exploited for cross-dialect information transfer. By merging these dialect experts into a single model, not only we mitigate the practical serving challenges, but we can improve the performance of each facilitating sharing of common knowledge.
As model merging approaches, we introduce our proposed method utilizing the Mixture of Experts (MoE) architecture.

##### MoE:

In a Mixture of Experts (MoE) model, each layer contains multiple MLP blocks, or “experts," and a router network selects specific experts to process each token at every position, combining their outputs. This architecture enhances the traditional MLP sub-layer within Transformer blocks by replacing it with multiple experts, each with its own set of parameters (Jiang et al., ; Fedus et al., ). MoE-based LLMs route tokens to different experts, increasing modeling expressiveness without significantly increasing the compute budget, as only a subset of experts is activated for each token. With different Transformer-based expert models, we can combine them into a single MoE model that leverages the expert-specific MLP layers. By initializing the router to select the corresponding expert for each token, we can combine the knowledge of expert models by activating multiple experts and merging the self-attention layers. This approach aligns well with our setting, where we have dialect-specific expert models that already have prior knowledge of SQL syntax. For dialect-specific keywords, we can use the router to select the appropriate MLP layer, which allows us to integrate three models into a single one without the need to train a new model from scratch. Figure 4 illustrates one Transformer block with the proposed method for constructing an MoE model from three distinct dialect expert models.

Figure: Figure 4: Our proposed method to initialize one Transformer block of a MoE model from different dialect experts, exemplified here for Postgres, SQLite, and BigQuery dialects to create an all in one model to address all. Objects in yellow demonstrate multi-dialect models
Refer to caption: extracted/5896809/figures/MoE.png

##### Multi-Dialect Routing:

An important aspect of the MoE framework is the routing (gating) mechanism. In MoE, the output for a given input
$x$ is computed as a weighted sum of the expert networks’ outputs, with the weights determined by the gating network (Jiang et al., ).
Given $n$ expert networks $\{E_{1},E_{i},...,E_{n}\}.$
The output is

$$ $\sum_{i=0}^{n-1}G(x)_{i}E_{i}(x),$ $$

where the gates’ outputs are determined based on the dot product of the input $x$ and the gate weights $W_{g}$ as follows:

$$ $G(x)=\text{Softmax}(TopK(dot(x,W_{g}))).$ $$

We propose to initialize these gates at each layer by averaging the hidden vectors of the dialect-specific keywords, derived from the training data of each model and based on the top $K$ most frequently occurring dialect-specific keywords from generated question-SQL pairs. This process begins by cataloging all dialect-specific keywords from our generated SQL queries, sorting them by frequency, and selecting the top-k keywords. These keywords are then processed by the model, where the hidden representations from the self-attention sub-modules for all tokens of these keywords are used to initialize the gates. The formula below provides the described method for initializing the gate weights:

$$ $W_{g}(i)=\frac{1}{K}\sum_{k=1}^{K}\sum_{j=1}^{k_{t}}h_{k_{j}},$ $$

where $W_{g}(i)$ is the ith column of the gate weight matrix corresponding to ith expert, $k_{t}$ is the number of tokens for the kth dialect keyword, and $h_{k_{j}}$ is the hidden representation of the j-th token of the k-th keyword. This approach increases the dot product between dialect-specific input keywords and their corresponding gate weight matrix columns, thereby boosting the weight for the dialect-specific expert. Although this method shows improved performance even without further training compared to other model merging approaches, we show superior joint modeling of the sub-models by further fine-tuning the MoE architecture on a mixed dataset from various dialects.

##### SLERP-based self-attention merging:

In our proposed approach, the MLP layers of each expert within the MoE model are initialized using the MLP sub-layers from models previously trained on distinct dialects. For the self-attention sub-layers of the MoE model, we employ Spherical Linear Interpolation (SLERP) (Goddard et al., ; Shoemake, ) to merge the initial weights of the self-attention layers (Key, Value, and Query projections) across multiple dialects. SLERP allows for smooth, non-linear transitions between two weight vectors while preserving the intrinsic geometric properties of the spherical space. The process begins by normalizing the weights of the Key, Value, and Query layers from different dialect models to unit magnitude, ensuring that they lie on the surface of a unit sphere. Once normalized, the angle ($\theta$) between the weight vectors is computed using the dot product. If the vectors are nearly col-linear (i.e., the dot product is close to 1), the merging process defaults to linear interpolation (LERP) for efficiency. Otherwise, SLERP estimates the scale factors based on the interpolation parameter
$t$ and the angular separation between the vectors:

$$ $\text{SLERP}(t,\mathbf{v}_{0},\mathbf{v}_{1})=\frac{\sin((1-t)\theta)}{\sin( \theta)}\mathbf{v}_{0}+\frac{\sin(t\theta)}{\sin(\theta)}\mathbf{v}_{1}.$ $$

Here, $v_{0}$ and $v_{1}$ represent the normalized weight vectors of the models. By merging the self-attention weights through SLERP, we can smoothly integrate the knowledge from different dialect-specific models into the initialization of the MoE model’s self-attention layers, providing a more effective starting point for model training.

## 3 Experiments

##### Datasets:

We use benchmark datasets tailored for three dialects. For SQLite, we use two datasets from BIRD: 1) the development set and 2) the mini development set. For PostgreSQL, we utilize three benchmarks: 1) BIRD queries transpiled to PostgreSQL, 2) BIRD PostgreSQL mini development set, and 3) Pagila—a dataset specific to PostgreSQL containing real-world queries originally written for PostgreSQL, which are extracted from online resources. For BigQuery, we use two datasets: 1) BIRD queries transpiled to BigQuery, and 2) the GitHub_repositories dataset, a public BigQuery dataset featuring BigQuery-specific sample question/SQL pairs obtained from tutorials and online resources. Further details of the datasets are provided in Section A.2.

##### Baselines:

In order to evaluate the quality of the synthetic queries generated with SQL-GEN, we compare the performance of the models trained on synthetic data considering the following datasets:
1) Gretel (Gretel, ): Gretel Text-to-SQL dataset consists $>$100K high-quality synthetic Text-to-SQL samples with a coverage across 100 distinct domains.
2) SQL create context (b mc2, ): This dataset consists $\sim$78K samples, obtained from the Spider (Yu et al., ) and WikiSQL (Zhong et al., ) datasets by cleaning these sources. All queries were created using a human-in-the-loop approach, providing a robust baseline for comparing data generated by LLMs to human-annotated data.
3) BIRD train set (Li et al., ): This dataset consists of $\sim$10k human-annotated samples. Queries are considered more complex in comparison to the Spider and WikiSQL benchmarks.

For dialects other than SQLite, all queries from the baselines are transpiled to the target dialects to ensure their validity. Details of the models and metrics are provided in Section A.3 and details of the seed templates and tutorials are provided in Section A.4.

### 3.1 Dialect-specific SQL keywords converge improvement

We compare our generated SQL queries with two baseline datasets in terms of the diversity of queries, focusing on the use of unique SQL keywords and the frequency of dialect-specific queries. The results are presented in Figure 5. For an equitable comparison, we sample 60K queries from each baseline. Since our Text-to-SQL dataset exclusively contains SELECT queries, we exclude samples that do not start with the SELECT keyword. According to the results, our dataset exhibits the highest diversity and the greatest number of dialect-specific queries compared to the baselines. Interestingly, the SQL create context dataset, which is intended to be a SQLite dataset, contains several queries using the STRUCT() keyword, which is supported by BigQuery, not SQLite.

Figure: Figure 5: Comparison between queries generated by our method with the baselines in terms of diversity of the SQL keywords and number of dialect-specific queries in each of them.
Refer to caption: extracted/5896809/figures/keyword_dist.png

### 3.2 PostgreSQL Results

For PostgreSQL, we train LoRA adapters for the CodeLlama 7B and Codestral 22B model on the transpiled baseline datasets and compared its performance against our proposed method. As shown in Table 1, our method achieves the highest performance on the PostgreSQL BIRD and Minidev benchmarks compared to other baselines, except for the BIRD train set. While training a model on the original BIRD training split delivers the highest performance on the BIRD development split, it significantly underperforms when evaluated on other PostgreSQL datasets, such as Pagila (as seen in the third row). This highlights the importance of diversity in training data to prevent overfitting to a specific distribution. In contrast, our approach achieves consistently high accuracy when evaluated on both the BIRD development split and other PostgreSQL datasets, demonstrating outstanding generalization ability. Moreover, these results demonstrate the importance of dialect specific datasets as the other transpiled queries couldn’t match the performance of our method.

**Table 1: Execution Accuracy (EX) of PostgreSQL Models on the three PostgreSQL benchmarks using CodeLlama 7B and Codestral 22B. "-" denotes the zero-shot performance of the models.**
| Training Dataset | Benchmark | Model | EX (%) | $\Delta$EX | Model | EX (%) | $\Delta$EX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Bird train set | PostgreSQL BIRD | CodeLlama 7B | 44.37 | +20.08 | Codestral 22B | 52.26 | +5.68 |
| Our synthetic dataset | PostgreSQL BIRD | CodeLlama 7B | 39.22 | +14.93 | Codestral 22B | 49.84 | 3.26 |
| Gretel Text-to-SQL | PostgreSQL BIRD | CodeLlama 7B | 28.05 | +3.76 | Codestral 22B | 40.55 | -6.03 |
| SQL Create Context | PostgreSQL BIRD | CodeLlama 7B | 13.35 | -10.94 | Codestral 22B | 36.17 | -10.41 |
| - | PostgreSQL BIRD | CodeLlama 7B | 24.29 | 0 | Codestral 22B | 46.58 | 0 |
| Bird train set | PostgreSQL Minidev | CodeLlama 7B | 31.0 | +17.8 | Codestral 22B | 36.0 | +4.2 |
| Our synthetic dataset | PostgreSQL Minidev | CodeLlama 7B | 25.4 | +12.2 | Codestral 22B | 33.0 | +2.2 |
| Gretel Text-to-SQL | PostgreSQL Minidev | CodeLlama 7B | 14.6 | +1.4 | Codestral 22B | 23.0 | -8.8 |
| SQL Create Context | PostgreSQL Minidev | CodeLlama 7B | 7.8 | -5.4 | Codestral 22B | 25.2 | -6.6 |
| - | PostgreSQL Minidev | CodeLlama 7B | 13.2 | 0 | Codestral 22B | 31.8 | 0 |
| Bird train set | Pagila | CodeLlama 7B | 19.56 | -4.35 | Codestral 22B | 43.47 | -6.53 |
| Our synthetic dataset | Pagila | CodeLlama 7B | 39.13 | +15.22 | Codestral 22B | 50 | 0.0 |
| Gretel Text-to-SQL | Pagila | CodeLlama 7B | 36.95 | +13.04 | Codestral 22B | 50 | 0 |
| SQL Create Context | Pagila | CodeLlama 7B | 8.69 | -15.22 | Codestral 22B | 36.95 | -13.05 |
| - | Pagila | CodeLlama 7B | 23.91 | 0 | Codestral 22B | 50 | 0 |

### 3.3 BigQuery Results

Similar to the PostgreSQL experiments, we present the results of the CodeLlama 7B and Codestral 22B model trained on various baseline datasets and evaluated on two BigQuery benchmark datasets: BIRD and the GitHub Repository database. Looking at the results provided in Table 2, consistent with the trends observed for the PostgreSQL dialect, on BigQuery BIRD, the model trained on our generated samples achieves the second highest performance, following the BIRD train set. For the GitHub Repository database, which is a BigQuery dialect-specific dataset, our model outperforms the second-best model by a 10% margin, further demonstrating the effectiveness of our method to train dialect specific models.

**Table 2: Execution Accuracy (EX) of BigQuery-specific models on the two BigQuery benchmarks using CodeLlama 7B and Codestral 22B. "-" denotes the zero-shot performance of the models.**
| Training Dataset | Benchmark | Model | EX (%) | $\Delta$EX | Model | EX (%) | $\Delta$EX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Bird train set | BigQuery BIRD | CodeLlama 7B | 38.04 | +21.47 | Codestral 22B | 47.74 | +9.55 |
| Our synthetic dataset | BigQuery BIRD | CodeLlama 7B | 33.53 | +16.96 | Codestral 22B | 47.24 | +9.05 |
| Gretel Text-to-SQL | BigQuery BIRD | CodeLlama 7B | 26.73 | +11.16 | Codestral 22B | 36.74 | -1.45 |
| SQL Create Context | BigQuery BIRD | CodeLlama 7B | 10.84 | -5.73 | Codestral 22B | 39.19 | +1.0 |
| - | BigQuery BIRD | CodeLlama 7B | 16.57 | 0 | Codestral 22B | 38.19 | 0 |
| Bird train set | Github Repository | CodeLlama 7B | 7.5 | -7.5 | Codestral 22B | 7.5 | -12.5 |
| Our synthetic dataset | Github Repository | CodeLlama 7B | 25.0 | +10.0 | Codestral 22B | 30.0 | +10.0 |
| Gretel Text-to-SQL | Github Repository | CodeLlama 7B | 17.5 | +2.5 | Codestral 22B | 22.5 | +2.25 |
| SQL Create Context | Github Repository | CodeLlama 7B | 0.0 | -15.00 | Codestral 22B | 20.0 | 0 |
| - | Github Repository | CodeLlama 7B | 15 | 0 | Codestral 22B | 20.0 | 0 |

### 3.4 SQLite Results

Utilizing SQL-GEN, we generate 20K samples for the SQLite dialect. Section A.4.2 studies the impact of the number of samples. We train three different models with different sizes from 7B to 22B on these samples. For a fair comparison with the baselines, we only use the Spider databases for generating the synthetic data. For this comparison, we train models on: 1) The entire BIRD training set; 2) 20K samples from the SQL Create Context (b mc2, ); and 3) 20K samples from the Gretel Text-to-SQL datasets (Gretel, ). We assess the Text-to-SQL performance of these models on the BIRD development set and minidev set (see Table 3). Additionally, we evaluate the zero-shot performance of each model and calculate the performance gains for each method relative to zero-shot.

SQL-GEN generated samples significantly surpass the Gretel dataset, achieving a large performance gain of approximately 10% across all model sizes.
Furthermore, LLMs trained on SQL-GEN synthetic data consistently outperform those trained on the human-annotated SQL Create Context data, underscoring the high quality and utility of the SQL-GEN synthetic data. While LLMs trained on the BIRD dataset consistently exhibit the highest performance on BIRD development sets, this outcome is likely due to overfitting to the canonical input distribution of the BIRD train set which is similar to its development set (Yu et al., ).

**Table 3: Execution Accuracy (EX) of SQLite Models on the BIRD development set and minidev set using CodeLlama 7B, CodeGemma 7B, and Codestral 22B Models. "-" denotes the zero-shot performance of the models.**
| Training Dataset | Model | Dataset | EX (%) | $\Delta$EX | Dataset | EX (%) | $\Delta$EX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Bird train set | CodeLlama 7B | dev set | 40.22 | +22.36 | minidev set | 38.4 | +24.8 |
| Our synthetic dataset | CodeLlama 7B | dev set | 38.33 | +20.47 | minidev set | 30.00 | +16.4 |
| Gretel Text-to-SQL | CodeLlama 7B | dev set | 26.01 | +8.15 | minidev set | 19.6 | +6.0 |
| SQL Create Context | CodeLlama 7B | dev set | 18.31 | +0.45 | minidev set | 12.6 | -1.0 |
| - | CodeLlama 7B | dev set | 17.86 | 0 | minidev set | 13.6 | 0.0 |
| Bird train set | CodeGemma 7B | dev set | 45.63 | +11.87 | minidev set | 40.4 | +10.4 |
| Our synthetic dataset | CodeGemma 7B | dev set | 42.37 | +8.64 | minidev set | 36.4 | +6.4 |
| Gretel Text-to-SQL | CodeGemma 7B | dev set | 30.83 | -2.93 | minidev set | 30.6 | +0.6 |
| SQL Create Context | CodeGemma 7B | dev set | 28.87 | -4.89 | minidev set | 29.6 | -0.4 |
| - | CodeGemma 7B | dev set | 33.76 | 0 | minidev set | 30.0 | 0.0 |
| Bird train set | Codestral 22B | dev set | 53.12 | +8.6 | minidev set | 50.4 | +10.0 |
| Our synthetic dataset | Codestral 22B | dev set | 50.45 | +5.93 | minidev set | 46.6 | +6.2 |
| Gretel Text-to-SQL | Codestral 22B | dev set | 37.87 | -6.65 | minidev set | 30.8 | -9.6 |
| SQL Create Context | Codestral 22B | dev set | 40.80 | -3.72 | minidev set | 36.8 | -3.6 |
| - | Codestral 22B | dev set | 44.52 | 0 | minidev set | 40.4 | 0.0 |

To pinpoint whether the gains are consistent across, we evaluate different models on different SQL query complexity levels: simple, medium, or challenging, as presented in Section A.5.

##### Database Adaptation:

SQL-GEN operates independently of specific databases, enabling the generation of high-quality synthetic data for any database. Therefore, as another use case of synthetic data, we introduce Database Adaptation, to improve the performance in cross-domain Text-to-SQL setting. This involves generating synthetic queries for databases for which no pre-existing question-SQL pairs are available. We apply this training in two distinct ways: (1) in-context learning, which leverages the generated queries directly within the model’s input context as demonstrations, and (2) model tuning, which involves supervised fine-tuning of the model weights:

##### Adaptation With Model Tuning:

Our synthetic data generation pipeline is designed to generate question-SQL pairs for any database. To demonstrate this, we generated 10K pure synthetic question-SQL pairs across the 11 databases in the BIRD development set and separately 10K samples for the entire databases in the BIRD training set using Gemini-1.5-pro. We then compared the performance of two model against a model trained on the original 10K training samples from the BIRD benchmark. The results are detailed in Table 5. The table indicates that our synthetic generation approach on the BIRD development set databases achieves performance comparable to the original BIRD training set with only 1.5% gap. This is particularly noteworthy given that generating synthetic samples is significantly less resource- and cost-intensive compared to creating 10,000 human-annotated samples. The latter involves 11 crowd-source workers to annotate the samples. Additionally, synthetic data generation on development split outperforms train split showing that SQL-Gen helps to learn unseen database and improve performance.

**Table 4: Using our proposed pipeline to generate 10K synthetic data for BIRD development set databases.**
| Training Data | EX (%) | $\Delta$EX |
| --- | --- | --- |
| BIRD train set | 40.22 | +22.36 |
| Synthetic sample on BIRD dev dbs | 38.78 | +20.92 |
| Synthetic sample on BIRD train dbs | 34.68 | +16.82 |
| Zero-shot (no training) | 17.86 | 0 |

##### Adaptation with In-context Learning:

An alternative method to enhance the performance of LLMs on task-specific datasets is through in-context learning (Brown et al., ). We explore the concept of database adaptation through in-context learning, using synthetic queries as few-shot in-context samples without additional model training. To evaluate this approach, we generate 500 synthetic samples for the California schools database from the BIRD development set. We then test the model’s performance on 89 samples from this database using different numbers of in-context samples. The results, presented in Table 5. For selecting the few-shot samples, we use cosine similarity between question embeddings. These results demonstrate that we can achieve a 10% improvement in accuracy without any training.

##### Data Augmentation:

Beyond merely creating a pool of pure synthetic question-SQL pairs for training, synthetic data generation offers the potential to augment existing datasets (e.g. mixing with original dataset), thereby enhancing model performance beyond what is achievable with solely the available data. We consider integrating synthetic data generated for specific target databases (as discussed in Database Adaptation, see Section 3.4) with pre-existing training datasets. To this end, we merge 10K synthetic question-SQL pairs generated on the BIRD development databases with the 10K pairs from the BIRD training set. We then train various models using this combined dataset and compared their performance to models trained solely on the original BIRD training set. For a balanced comparison, models using the combined datasets are trained for only one epoch, whereas those trained exclusively on the BIRD training set are trained fro two epochs. As shown in Table 6, augmenting training data results in a performance improvement of up to 5.6%, a significant enhancement compared to previous work, such as Yang et al. () (which demonstrated only a 1.5% improvement in performance after augmentation on the same base model CodeLLama 7B).

**Table 6: Performance comparison of the data augmentation method on the BIRD development set using different LLMs.**
| Training Dataset | Model | EX (%) | $\Delta$EX |
| --- | --- | --- | --- |
| BIRD train set | CodeLlama 7B | 40.22 | 0 |
| BIRD Train + synthetic | CodeLlama 7B | 45.82 | +5.6 |
| BIRD train set | CodeGemma 7B | 45.63 | 0 |
| BIRD Train + synthetic | CodeGemma 7B | 51.10 | +5.47 |
| BIRD train set | Codestral 22B | 53.12 | 0 |
| BIRD Train + synthetic | Codestral 22B | 56.45 | +3.33 |

### 3.5 Experts Merging Results

We evaluate various expert merging approaches for integrating dialect-specific models into a single unified model and compare to the proposed approach based on the Mixture of Experts (MoE) architecture. We utilize three expert CodeLlama 7B models, each trained on synthetic question-SQL pairs for SQLite, PostgreSQL, and BigQuery. We consider three popular model merging techniques: DARE (Yu et al., ), TIES (Yadav et al., ), and SLERP. Unlike the first two, SLERP can only merge two models at a time. Therefore, we initially merge the SQLite and PostgreSQL experts and then combined the resulting model with the BigQuery expert. Additionally, we fine-tune a generalist (not dialect-specific) CodeLlama 7B and MoE 3x7B model on 40K samples from a mix of different dialects to establish a baseline for comparison. The generalist MoE baseline is a MoE 3x7B model initialized from CodeLlama 7B model and trained with 40K combined dialect samples for 1 epoch. We compare all these methods to the proposed method for initializing the MoE model which is trained only for one epoch of 20K samples from different dialects. We train for a single epoch to promote effective collaboration among the submodules. We also include the performance of the proposed MoE model before the single epoch fine-tuning to better understand understand the effectiveness of the proposed initialization method. We assess the performance of the different models on PostgreSQL’s Pagila, BigQuery’s Github Repository, and 10% of random samples from the SQLite BIRD dev set, with results detailed in Table 7. The table shows that the proposed MoE model outperforms other models, even surpassing the individual dialect experts. This highlights the effectiveness of our approach in sharing common SQL knowledge across dialects while maintaining dialect-specific expertise. The MoE architecture also enhances the model’s learning capacity, contributing to improved overall performance. Notably, our initialization method is more effective at maintaining high dialect-specific performance compared to the generalist MoE 3x7B model. Among all merging techniques, SLERP achieves the highest performance, surpassing even the generalist model trained on the combined dialect-specific datasets, which is the main reason for initializing the self-attention sub-layers. Moreover, the results suggest that our proposed method for initialization even before fine-tuning provides a strong baseline, surpassing TIES and DARE methods for model merging. In Section A.8, we provide detailed analysis of token-level routing for MoE architecture.

**Table 7: Comparison between different dialect expert merging approaches and our proposed MoE for dialect benchmarks. Generalist model refers to CodeLlama trained on the combination of the dialect datasets.**
| Model | BIRD SQLite | PostgreSQL Pagila | BigQuery BIRD | Overall |
| --- | --- | --- | --- | --- |
| CodeLlama 7B SQLite expert | 34.01 | 39.13 | 25 | 32.71 |
| CodeLlama 7B Postgres expert | 29.93 | 39.13 | 20 | 29.68 |
| CodeLlama 7B BigQuery expert | 33.33 | 32.60 | 27.5 | 31.14 |
| CodeLlama 7B generalist | 33.33 | 32.66 | 32.25 | 32.83 |
| Merged experts + SLERP | 34.69 | 39.13 | 27.5 | 33.77 |
| Merged experts + TIES | 35.37 | 30.43 | 27.5 | 31.1 |
| Merged experts + DARE | 35.37 | 36.95 | 17.5 | 29.94 |
| MoE 3x7B (ours) | 36.05 | 39.13 | 22.5 | 32.56 |
| MoE 3x7B fine-tuned (ours) | 34.69 | 39.13 | 32.25 | 35.44 |
| MoE 3x7B generalist | 34.01 | 41.3 | 25 | 33.43 |

## 4 Conclusions

This work addressed the challenge of cross-dialect generalization in Text-to-SQL by introducing SQL-GEN, a novel framework for generating high-quality synthetic training data from dialect-specific tutorials. The proposed framework addresses the unique challenges such as keywords and functions being different for each SQL dialect, constituting a scalable approach. It significantly narrows the performance gap with human-annotated datasets and creates the highest quality datasets for other dialects. Our comprehensive evaluations across three models and multiple benchmarks showcase the effectiveness of the proposed data generation framework. To further enhance cross-dialect performance, we proposed a novel Mixture-of-Experts (MoE) initialization method that leverages shared knowledge across dialects. By merging self-attention layers from dialect-specific models and initializing expert gates with dialect-specific keywords, we created a unified model capable of outperforming single-dialect models. Our findings demonstrate the effectiveness of SQL-GEN in generating high-quality synthetic data for diverse SQL dialects and highlight the potential of our MoE initialization method for building robust, multi-dialect Text-to-SQL systems. This work opens avenues for future research in cross-lingual Text-to-SQL and the application of synthetic data generation to other code generation tasks.

## Appendix A Appendix

### A.1 Related Work

#### A.1.1 Synthetic Data Generation

Early work for data augmentation for Text-to-SQL largely rely on human annotations to verify the generated SQL queries or extract high-quality question-SQL pairs (Iyer et al., ; Yu et al., ). Guo et al. () use a pattern-based approach to generate SQL queries and utilize a copy-based Seq2Seq model to directly translate SQL queries into natural language questions. Some of the recent methods (Wu et al., ; Yu et al., ; Zhao et al., ; Wang et al., ) rely on grammar-based approaches to generate question-SQL pairs. Wu et al. () use an abstract syntax tree grammar to generate SQL queries and then employs a hierarchical SQL-to-question generation model to obtain questions for the SQL queries. Similarly, Yu et al. () extract and manually annotate question and SQL templates from Spider (Yu et al., ) to induce a grammar, then use the grammar to generate synthetic samples for databases in Spider and WikiTables (Bhagavatula et al., ). However, all methods relying on grammars have the drawback of generating samples that lack diversity and highly depend on the grammar used (Yu et al., ), which makes them not suitable for tasks that require generalization to new schemas.

Recently, Li et al. () propose a bidirectional method with question-to-SQL and SQL-to-question augmentation. In the former, they use some human-annotated samples with in-context learning with LLMs to generate queries for a new database, and in the latter, they extract templates from Spider and fill those templates with the schema of a given database. This method has the limitation that the diversity of the question and SQL pairs is restricted to either templates or in-context samples. Concurrently with our work, SENSE (Yang et al., ) proposed a two-step synthetic data generation process to enhance the performance of open-source text-to-SQL models. In the first step, they utilize a robust LLM to generate a supervised fine-tuning dataset with a single LLM call. In the second stage, they employ a smaller, weaker LLM to produce some incorrect SQL queries, which are then used to construct a preference dataset. The initial phase of their method is similar to our proposed approach; however, their method’s simplicity, which lacks execution result filtering or conditioning on externally provided SQL keywords and relies solely on the LLMs’ parametric knowledge, contrasts with our method that incorporates external knowledge to craft diverse queries. Lastly, Gretel () release a high-quality large dataset of 100K question-SQL pairs from different domains.(^1^11The methodology to generate the pairs is not publicly available.) Overall, none of the previously mentioned approaches consider different dialects and they are proposed for SQLite (^2^22Gretel dataset doesn’t specify the dialect.), which is a significant drawback of their work.

In the domain of synthetic data generation for code, recent work such as Reflexion (Shinn et al., ) leverage external or internal feedback signals to enhance the code reasoning capabilities of language models. Code Alpaca features a dataset of 20K code instructions automatically generated by applying SELF-INSTRUCT (Wang et al., ) to LLMs across different seed tasks. WizardCoder (Luo et al., ) introduces Code Evol-Instruct, which uses manually crafted prompts to guide LLMs, thereby increasing the complexity and diversity of the synthetic data. Similarly, Magicoder (Wei et al., ) proposes OSS-INSTRUCT, which consists 75K diverse synthetic instruction samples from open-source code snippets that are used as the seeds to both increase diversity and also control the data generation process.

#### A.1.2 Model Merging

Training specialized, task-specific models presents several challenges, including the storage costs associated with maintaining multiple models, the substantial memory requirements for deploying these models, and the rapid obsolescence of models as training datasets age. One proposed solution to mitigate these issues is model merging (Goddard et al., ). Initial approaches to model merging, such as Task Arithmetic (Ilharco et al., ), involve calculating task-specific vectors by determining the weight differences between the fine-tuned model and its base counterpart. These vectors are then linearly combined and reintegrated with the original base model. Subsequent methodologies like DARE, TIES, and Model BreadCrumbs (Yadav et al., ; Yu et al., ; Davari & Belilovsky, ) have aimed to minimize interference among task-specific models through techniques such as sparsification, sign consensus algorithms, and the exclusion of extreme values. Additionally, DARE introduces random pruning to align more closely with the base model’s performance (Goddard et al., ). More recently, the integration of model merging with Mixture of Experts (MoE) architectures has been explored. This method, termed FrankenMoEs, initializes MoE MLP layers using weights from task-specific models (Goddard, ; Tang et al., ). Our work extends these efforts by specifically leveraging features from dialect-specific models for gate initialization and merging self-attention sublayers within transformer architectures.

### A.2 Datasets Details

#### A.2.1 SQLite

To the best of our knowledge, the majority of large-scale, cross-domain Text-to-SQL datasets are tailored for the SQLite dialect. Among these, the Spider (Yu et al., ) and BIRD Li et al. () datasets are two popular benchmarks used to evaluate Text-to-SQL model performance (Pourreza & Rafiei, ; Wang et al., ; Talaei et al., ; Li et al., ), establishing them as primary standards in this area. We use the Spider training set to derive seed templates. To ensure a fair comparison, we report the results using the BIRD benchmark for the SQLite dialect, with the Spider dataset serving as a baseline to assess the quality of our synthetic samples. The BIRD benchmark includes two development sets: the original dev set, which contains 1534 question-SQL pairs with some incorrect SQL queries Li et al. (), and the minidev set, which features smaller size of 500 higher quality question-SQL pairs. We evaluate on both.

#### A.2.2 PostgreSQL

As mentioned in the previous section, there is a shortage of human-annotated benchmarks for dialects other than SQLite. Therefore, for PostgreSQL dialect, we use the following datasets to compare the performance of the models:

##### PostgreSQL BIRD:

All 11 databases in the BIRD development set are migrated from SQLite to PostgreSQL, and their SQL queries are transpiled to PostgreSQL using Mao (). This migration and transpilation are conducted under a best-effort setting. However, some challenges are encountered: a few databases have foreign key violations, and some queries cannot be successfully transpiled to PostgreSQL. Out of the 1534 samples in the development set, 951 queries are successfully migrated for PostgreSQL.

##### PostgreSQL MiniDev:

Similar to the approach we use for the PostgreSQL BIRD dataset, the authors of BIRD transpile queries in the minidev set, manually annotating any pairs that cannot be directly translated from SQLite to PostgreSQL. This dataset comprises 500 question-SQL pairs.

##### Pagila:

Since the BIRD benchmark was originally developed for SQLite, the transpiled queries do not utilize many PostgreSQL-specific functions and keywords. To address this, we created a PostgreSQL-specific benchmark, Pagila (). The Pagila database mimics a real-world business by modeling a DVD rental store. It includes tables for films, actors, customers, inventory, rental transactions, and more, making it a useful resource for educational purposes. This database is designed to provide a standard schema for use in books, tutorials, and articles. We gathered a dataset of 46 human-annotated question-SQL pairs, which were validated and extracted from open-source resources for this database.

#### A.2.3 BigQuery

We use the following baselines for reporting the performance for BigQuery dialect:

##### BigQuery BIRD:

Similar to the approach mentioned for PostgreSQL, all 11 databases in the BIRD development set are migrated from SQLite to BigQuery, and their SQL queries are transpiled to BigQuery using Mao (). Out of the 1534 samples in the development set, 1309 queries are successfully migrated for BigQuery.

##### Github Repositories:

In our work, for the BigQuery-specific database, we utilized one of the publicly available and widely used databases, the GitHub repositories (). This database allows for monitoring and analyzing GitHub’s activity since 2011. We gathered a dataset of 40 human-annotated question-SQL pairs, validated and extracted from open-source resources for this database.

### A.3 Models & Metrics

#### A.3.1 Models

To evaluate the quality of the generated samples, we fine-tune models from different families, including CodeLlama 7B (Roziere et al., ), CodeGemma 7B (), and Codestral 22B (Mistral, ), using LoRA adapters for all linear layers (Hu et al., ) with a rank of 128 and alpha of 256. For the synthetic data generation process we use Gemini 1.5 pro as the main model and Gemini 1.5 flash as the quality check model. To ensure the data generation process is affordable and replicable, we also include high-performing, open-source LLMs for synthetic data generation. For template expansion and filling, we employ Llama-3-70B (), and for quality check step, we employ Mixtral-8x7B (Jiang et al., ).

#### A.3.2 Metrics

We primarily focus on execution accuracy (EX) as the main metric, which is widely accepted as the standard for all Text-to-SQL benchmarks (Yu et al., ; Li et al., ).

### A.4 Method Seeds

In this section, we present details regarding the number of seed SQL templates extracted from the Spider train set, which comprises 8,659 training examples across 146 databases. To generate seed templates for dialects other than SQLite, we transpiled the queries from SQLite to the target dialects using SQLGlot. Table 8 provides the counts of seed SQL queries for each dialect. Moreover, for scraping the tutorials we used the following websites for each dialect:

- •
SQLite: [SQLite tutorial](https://www.javatpoint.com/sqlite-tutorial)
- •
PostgreSQL: [PostgreSQL tutorial](https://www.javatpoint.com/postgresql-tutorial)
- •
BigQuery: [BigQuery syntax](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax)

**Table 8: Number of seed SQL templates extracted from the Spider training dataset for three dialects of SQLite, BigQuery, PostgreSQL.**
| Dialect | Number of templates |
| --- | --- |
| SQLite | 1458 |
| BigQuery | 1665 |
| PostgreSQL | 1293 |

#### A.4.1 Quality Check Ablation

In our proposed method, we opted to use a secondary LLM to act as a judge in the quality check step, ensuring the high quality of the generated samples and avoiding repetition of previous errors. In this section, we assess this approach by comparing two scenarios: one where the same LLM acts as judge, and another where a secondary LLM performs the judging role. The results, presented in Table 9, demonstrate that the CodeLlama 7B model trained on the dataset filtered by a secondary model achieved higher performance on the BIRD development set, thus validating our strategy.
Moreover, Table 10 provides the result of removing the quality check step and shows a performance drop in accuracy, validating the importance of this step to remove low quality samples.

**Table 9: Performance comparison between two scenarios, when the same model generates and filters candidate samples, and another when a secondary model is used for filtering.**
| Base Model | Judge Model | EX (%) |
| --- | --- | --- |
| Mixtral 8x7B | Mixtral 8x7B | 32.59 |
| Llama 3 70B | Llama 3 70B | 33.41 |
| Llama 3 70B | Mixtral 8x7B | 34.55 |

**Table 10: Performance on the ablation of the quality checker model with Codellama 7B on BIRD dev set. OS refers to using open-source models like Llama3 and Mixtral for data generation.**
| Pipeline | EX (%) |
| --- | --- |
| Pipeline without quality check (OS) | 32.85 |
| Full pipeline (OS) | 34.55 |

#### A.4.2 The Impact of the Sample Size

Due to the limited availability of large-scale benchmarks for dialects other than SQLite, our ablation studies focus solely on the SQLite dialect. For each target dialect, we use our method to generate 20K samples. We assess the impact of varying sample sizes on the final performance of the model. Table 11 presents the performance with the CodeLlama 7B model when trained on different sample sizes generated by Llama 3 and Mixtral models, and tested on the BIRD development set. As indicated, there is diminishing return in performance as the sample size increases.

**Table 11: Evaluating the performance of CodeLlama 7B using different sample sizes on BIRD development set. "-" denotes the zero-shot performance of the model**
| #Samples | Model | EX (%) | $\Delta$EX |
| --- | --- | --- | --- |
| - | CodeLlama | 17.86 | 0 |
| 5000 | CodeLlama | 32.59 | + 14.73 |
| 10000 | CodeLlama | 33.57 | +15.71 |
| 20000 | CodeLlama | 34.55 | +16.69 |

### A.5 Complexity Analysis

For complexity analysis we used official BIRD classification based on the number and type of the SQL keywords used in the ground truth SQL query for each question in BIRD development set. The results are provided in the Table 12 for the two synthetic and human annotated baselines together with the zero-shot performance of CodeLlama 7B model. Based on the results model trained on our synthetic data has the highest performance across all of the complexity levels. Interstingly, due to the simplisity of the samples in the SQL create context dataset performance on the challenging samples is even lower than the zero-shot baseline.

**Table 12: Comparison of different datasets across varying SQL query complexities on the BIRD development set for CodeLlama 7B trained on each dataset. "-" denotes the zero-shot performance of the models**
| Training Dataset | Model | simple EX (%) | moderate EX (%) | challenging EX (%) |
| --- | --- | --- | --- | --- |
| Our synthetic dataset (Gemini) | CodeLlama | 49.51 | 24.08 | 12.5 |
| Gretel Text-to-SQL | CodeLlama | 31.78 | 11.61 | 11.8 |
| SQL Create Context | CodeLlama | 25.18 | 9.67 | 2.08 |
| - | CodeLlama | 24.1 | 7.95 | 9.72 |

### A.6 Sample Generation Filters

#### A.6.1 execution check

Unlike the template expansion step, SQL queries in this stage are generated from actual databases, allowing us to execute the queries over the databases. This capability enables us to utilize dialect-specific database engines to discard samples that are syntactically incorrect. This method is more robust than the parsing checks with SQLglot, as employed in Gretel (), providing a more effective way to ensure the accuracy of our SQL queries.

#### A.6.2 Question-SQL Mismatch

During the query generation process using various LLMs such as Gemini (Team et al., ), GPT-3.5 Turbo, and Llama-3-70B (), we observe a recurring issue where some mismatches occurred between the conditions in the generated SQL queries and the corresponding questions. To minimize these mismatches, we develope a set of validator functions to detect inconsistencies. For each generated SQL query, we extract all conditions that correspond to database values using a SQL parser. We then calculate the maximum semantic similarity and the minimum edit distance between these conditions and all keywords in the question. SQL queries where a keyword’s minimum distance exceeds a threshold $\beta_{1}$ or whose maximum semantic similarity is below another threshold $\beta_{2}$ are discarded. Figure 6 illustrates an example of question-SQL pair which is rejected because of mismatch between questions and SQL outputs.

Figure: Figure 6: An example of a filtered question-SQL pair due to question and SQL mismatch.
Refer to caption: extracted/5896809/figures/sample_filtering.png

#### A.6.3 Aggregation Check

Another consistent issue with the LLMs was the inappropriate use of aggregation functions on columns that already contain aggregated values. For example, in response to the question, “What are the average ages of singers?" the LLM might generate: “SELECT AVG(average_age) FROM singer", where there is a redundant aggregation function. To address these cases, we examine the SQL queries for aggregation functions. If the column name already includes an aggregation function in its name, we discard those queries.

#### A.6.4 Deduplication And Length Check

Similar to the approaches proposed in Wei et al. (); Wang et al. (), we discard duplicated SQL queries and pairs where the question length exceeds a specific threshold, $\alpha_{1}$.

### A.7 Dialect Specific Keywords

This section presents some examples of dialect-specific keywords for BigQuery, PostgreSQL, and SQLite. These keywords, listed in Table 13, are not supported interchangeably among the three dialects. These keywords are just samples of dialect specific keywords and there are many more dialect specific keywords and functions.

**Table 13: List of some of SQL keywords that are not supported entirely across all three dialects of BigQuery, PostgreSQL, and SQLite.**
| Keyword | SQLite | PostgreSQL | BigQuery |
| --- | --- | --- | --- |
| CREATE MODEL | $\times$ | $\times$ | $\checkmark$ |
| ML.TRANSLATE | $\times$ | $\times$ | $\checkmark$ |
| ML.GENERATE_TEXT | $\times$ | $\times$ | $\checkmark$ |
| ML.ANNOTATE_IMAGE | $\times$ | $\times$ | $\checkmark$ |
| SAFE | $\times$ | $\times$ | $\checkmark$ |
| QUALIFY | $\times$ | $\times$ | $\checkmark$ |
| WITH OFFSET | $\times$ | $\times$ | $\checkmark$ |
| ARRAY_AGG | $\times$ | $\checkmark$ | $\checkmark$ |
| STRUCT | $\times$ | $\times$ | $\checkmark$ |
| ILIKE | $\times$ | $\checkmark$ | $\times$ |
| LATERAL | $\times$ | $\checkmark$ | $\times$ |
| SERIAL | $\times$ | $\checkmark$ | $\times$ |
| CTID | $\times$ | $\checkmark$ | $\times$ |
| PRAGMA | $\checkmark$ | $\times$ | $\times$ |
| REGEXP_CONTAINS | $\times$ | $\times$ | $\checkmark$ |
| REGEXP_MATCHES | $\times$ | $\checkmark$ | $\times$ |
| GLOB | $\checkmark$ | $\times$ | $\times$ |
| JULIANDAY | $\checkmark$ | $\times$ | $\times$ |
| DATE_TRUNC | $\times$ | $\checkmark$ | $\times$ |
| TIMESTAMP_TRUNC | $\times$ | $\times$ | $\checkmark$ |

### A.8 MoE Analysis

We analyze the hidden representations of our proposed MoE model and compare it with the baseline generalist MoE model across three distinct layers: Layer 1, Layer 16, and Layer 32. Although both MoE models are trained with a load balancing loss, our initialization approach for the gates leads to an expert collapse in the middle layers. This issue primarily stems from the high similarity in the hidden representations of the positive prompts for each dialect expert across all layers. Additionally, similar to the experiments conducted with the original Mixtral model (Jiang et al., ), there is no distinct expert associated with different token types in either our MoE model or the baseline generalist MoE model.

Figure: Figure 7: Token routing for the MoE model, initialized from CodeLlama and trained on a 40K samples dataset. The top figure illustrates Layer 1, the middle figure shows Layer 16, and the bottom figure corresponds to Layer 32.
Refer to caption: extracted/5896809/figures/MoE-baseline.jpg

Figure: Figure 8: Token routing for our proposed method for initializing the MoE model, and trained on a 20K samples dataset. The top figure illustrates Layer 1, the middle figure shows Layer 16, and the bottom figure corresponds to Layer 32.
Refer to caption: extracted/5896809/figures/Our-MoE.jpg

### A.9 Prompt Templates

This section provides the detailed prompts used in this work for each of the step in our work.

#### A.9.1 Template Expansion

This section provides the prompt for template expansion step, Figure 9, where a seed template together with a sampled dialect-specific tutorial is passed to the LLM and asked to generate a new dialect-specific template. Additionally Figure 3 provides an example of template expansion for BigQuery dialect.

Figure: Figure 9: Prompt used for the template expansion step (PTempGen)

#### A.9.2 Sample Generation

In this section, we provide the prompt for the sample generation step, Figure 10, where a dialect-specific template together with a database schema are passed to the LLM and asked to generate question/SQL pair. Additionally Figure 11 provides an example of sample generation step.

Figure: Figure 10: Prompt used for the sample generation (PGen)

Figure: Figure 11: An example of sample generation using a random database and sampled SQL template.
Refer to caption: extracted/5896809/figures/sample_generation_example.png

#### A.9.3 Quality Check

This section outlines the template for the quality check prompt, Figure 12. The template receives a database schema, a generated question, a generated SQL query, and the execution result of the query. It then identifies and resolves any semantic discrepancies between the pair.

Figure: Figure 12: Prompt used for the Quality Check step (PQuality)

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: No more questions. Please see above.
soundness: 2
strengths: 1. The method is evaluated on three SQL dialects and proves its effectiveness.
2. The author has built a text-to-sql dataset that includes SQL dialects. If this dataset is open source, it will be of certain value to research.
summary: This paper introduces the SQL-GEN framework, which bridges the dialect gap for text-to-SQL systems by generating high-quality synthetic data for any dialect and merging dialect-specific models into a unified model using a novel Mixture of Experts (MoE) initialization method. The framework significantly improves execution accuracy on unseen real-world multi-dialect benchmarks and reduces the gap compared to large-scale human-annotated data.
weaknesses: 1. The paper presents contributions that, while valuable, appear to have limited novelty compared to existing work in the field[1].
2. In addition, from the results, the improvement of synthetic data is not as good as that of bird train set.

[1] Yu T, Wu C S, Lin X V, et al. Grappa: Grammar-augmented pre-training for table semantic parsing[J]. arXiv preprint arXiv:2009.13845, 2020.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - On a side note, it would be interesting to consider cross-lingual transfer methods for text-to-SQL as seen in multilingual NLP tasks, which might provide a useful baseline for comparison.
soundness: 3
strengths: - **Originality**
    - The task of addressing dialects in text-to-SQL systems is nascently explored, and this paper makes a valuable contribution by introducing a novel MoE approach to manage multi-dialect scenarios effectively.
- **Quality and Clarity**: 
     - The paper is generally easy to follow.
     - The experiments comprehensively test SQL-GEN across multiple benchmarks and datasets and validate the performance.
summary: In the well-studied text-to-SQL task, the authors examine handling multiple SQL dialects and propose SQL-GEN, a three-step method to adapt text-to-SQL models trained on one SQL dialect to other dialects by generating synthetic text-to-SQL samples. They also introduce a Mixture of Experts approach to merge dialect-specific SQL models into a single model for efficient cross-dialect knowledge sharing and reducing maintenance costs by selectively activating dialect-specific experts based on the input. The method is tested on BIRD and Paglia benchmarks and is evaluated against baseline datasets to validate SQL-GEN's efficacy.
weaknesses: - Figures, especially Figures 3 and 4, are somewhat dense and challenging to interpret. A clearer visual representation or more detailed explanations would enhance accessibility.
- The text in Figure 5 could be enlarged to improve readability, as it currently appears difficult to see.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does SQL-GEN handle dialect-specific syntax that drastically changes query logic, such as proprietary SQL functions that are unique to one dialect? It would be better to provide specific examples of how SQL-GEN handles a few key proprietary functions from different dialects, and discuss any limitations in this area.
2. What measures are in place to ensure the synthetic queries generated by SQL-GEN align closely with real-world dialect usage beyond benchmarks?
soundness: 3
strengths: 1. SQL-GEN’s synthetic data generation effectively addresses the lack of annotated data for various SQL dialects, enhancing Text-to-SQL systems' adaptability across dialects without needing human annotations. And it is the first work to discuss its important problem.
2. The moe architecture efficiently merges dialect-specific models into a unified framework, promoting knowledge transfer and improving performance across dialects, making the system resource-efficient for real-world use.
3. SQL-GEN enhances execution accuracy by up to 20% over previous methods, making Text-to-SQL models more reliable for multi-dialect databases, with potential performance improvements in specific dialects like PostgreSQL and BigQuery.
4. By reducing the dependency on human annotations and enabling effective multi-dialect handling, SQL-GEN offers a cost-efficient approach to multi-dialect Text-to-SQL training. It would be better to open-source the generative model to help the community.
summary: The paper addresses the challenges of adapting Text-to-SQL systems across SQL dialects like BigQuery, PostgreSQL, and SQLite. Traditional Text-to-SQL models, optimized for SQLite, struggle with dialect-specific syntax variations, making translation and model adaptability challenging. SQL-GEN proposes a synthetic data generation method, SQL-GEN, which leverages dialect-specific SQL tutorials to create diverse, high-quality training samples for any SQL dialect. It introduces a Mixture of Experts (MoE) approach for merging dialect-specific models, enhancing multi-dialect capability.
weaknesses: 1. While SQL-GEN aims to create a unified model that bridges dialect gaps, merging dialect-specific models can dilute the depth of expertise for each dialect, leading to a possible trade-off between dialect-specific expertise and generalizability. It would be better to discuss the drawback of SQL-GEN in each dialect-specific example and provide a detailed analysis of performance on dialect-specific features or edge cases for each dialect. This could help clarify if and where any expertise dilution occurs.

2. The Mixture of Experts (MoE) approach enables the merging of dialect-specific models, but it also requires substantial computational resources for training, fine-tuning, and deploying models, especially as more dialects are added. It would be better to incorporate an easy SFT baseline to compare and provide a computational resource comparison between their MoE approach and a standard fine-tuning approach for multiple dialects. This would give readers a clearer picture of the trade-offs involved.

Meta-review: This paper presents a method for generating synthetic data for text-to-SQL tasks in multiple "dialects" (e.g., BigQuery, PostgreSQL, SQLite). This is done by generating abstract queries which are then refined into each dialect.  Training on this data produces effective models for each setting. These models are then merged by initializing an MoE model with each expert using a novel procedure. Results across several benchmarks compare training on the synthetic data from this work to training on existing datasets such as BIRD, as well as the performance of the MoE model versus individual expert models.

This paper presents its problem setting very clearly, proposes a clear solution, and evaluates that solution very thoroughly. The mixture-of-experts approach is an interesting way to combine models.

The main weaknesses of the work are the model performance. Comparing to training on this paper's synthetic data to training on BIRD, and comparing MoE and SFT, the results are mixed (though generally positive in favor of this paper). The authors assert that their approach provides some conceptual advantages in both settings (e.g., generality compared to BIRD-trained models, where in-domain test sets are artificially easy). However, it's not clear to me whether the needle has been substantially moved in this domain (i.e., whether a broad range of practitioners in text-to-SQL really need to pay attention to these results), and also whether there's a generalizable takeaway method for a typical attendee of ICLR.

Overall, this paper delivers data and systems that will be useful to practitioners in this area, but I'm not sure these will be as useful or interesting to the broader ICLR audience.

Decision: Reject