SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging

Authors: Mohammadreza Pourreza, Ruoxi Sun, Hailong Li, Lesly Miculicich, Tomas Pfister, Sercan O Arik
Primary area: foundation or frontier models, including LLMs
Keywords: Text-to-SQL, LLM, Synthetic Data Generation, Databases

Abstract:
Text-to-SQL systems, that convert natural language queries into SQL programs, have seen significant progress with recent breakthroughs.  However,  these have been primarily for the SQLite dialect and adapting Text-to-SQL systems to other SQL dialects like BigQuery and PostgreSQL remains a challenge due to the diversity  in  SQL  syntaxes  and  functions,  along  with  the  high  cost  of  collecting and curating SQL-specific training data.   To this end,  we introduce SQL-GEN,a framework for generating high-quality synthetic data for any dialect guided by dialect-specific tutorials.  We demonstrate the effectiveness of SQL-GEN in creating training data to significantly improve the downstream Text-to-SQL performance for other dialects – it improves the execution accuracy by up to 20% over previous  methods,  and  reduces  the  gap  with  large-scale  human-annotated  data on unseen real world multi-dialect benchmarks.  Moreover, combining our synthetic data with human-annotated data provides additional performance boosts up to 5.6%. Towards unifying the multi-dialect capability in a single system, we also introduce a novel Mixture of Experts (MoE) initialization method that integrates dialect-specific models by merging self-attention layers and initializing the gates with dialect-specific keywords, yielding one unified and versatile model adept for multiple SQL dialects,  further enhancing performance across different SQL dialects. By leveraging shared core features of multiple dialect-specific models, our MOE demonstrated superior performance compared with models trained on individual dialects alone

Decision: reject
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730467922150

thread_type:
official_review

<thread_object>
creator: reviewer_a2fh

<content>
summary:
The paper addresses the challenges of adapting Text-to-SQL systems across SQL dialects like BigQuery, PostgreSQL, and SQLite. Traditional Text-to-SQL models, optimized for SQLite, struggle with dialect-specific syntax variations, making translation and model adaptability challenging. SQL-GEN proposes a synthetic data generation method, SQL-GEN, which leverages dialect-specific SQL tutorials to create diverse, high-quality training samples for any SQL dialect. It introduces a Mixture of Experts (MoE) approach for merging dialect-specific models, enhancing multi-dialect capability.

strengths:
1. SQL-GEN’s synthetic data generation effectively addresses the lack of annotated data for various SQL dialects, enhancing Text-to-SQL systems' adaptability across dialects without needing human annotations. And it is the first work to discuss its important problem.
2. The moe architecture efficiently merges dialect-specific models into a unified framework, promoting knowledge transfer and improving performance across dialects, making the system resource-efficient for real-world use.
3. SQL-GEN enhances execution accuracy by up to 20% over previous methods, making Text-to-SQL models more reliable for multi-dialect databases, with potential performance improvements in specific dialects like PostgreSQL and BigQuery.
4. By reducing the dependency on human annotations and enabling effective multi-dialect handling, SQL-GEN offers a cost-efficient approach to multi-dialect Text-to-SQL training. It would be better to open-source the generative model to help the community.

weaknesses:
1. While SQL-GEN aims to create a unified model that bridges dialect gaps, merging dialect-specific models can dilute the depth of expertise for each dialect, leading to a possible trade-off between dialect-specific expertise and generalizability. It would be better to discuss the drawback of SQL-GEN in each dialect-specific example and provide a detailed analysis of performance on dialect-specific features or edge cases for each dialect. This could help clarify if and where any expertise dilution occurs.

2. The Mixture of Experts (MoE) approach enables the merging of dialect-specific models, but it also requires substantial computational resources for training, fine-tuning, and deploying models, especially as more dialects are added. It would be better to incorporate an easy SFT baseline to compare and provide a computational resource comparison between their MoE approach and a standard fine-tuning approach for multiple dialects. This would give readers a clearer picture of the trade-offs involved.

questions:
1. How does SQL-GEN handle dialect-specific syntax that drastically changes query logic, such as proprietary SQL functions that are unique to one dialect? It would be better to provide specific examples of how SQL-GEN handles a few key proprietary functions from different dialects, and discuss any limitations in this area.
2. What measures are in place to ensure the synthetic queries generated by SQL-GEN align closely with real-world dialect usage beyond benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601493454

thread_type:
official_review

<thread_object>
creator: reviewer_vtyn

<content>
summary:
In the well-studied text-to-SQL task, the authors examine handling multiple SQL dialects and propose SQL-GEN, a three-step method to adapt text-to-SQL models trained on one SQL dialect to other dialects by generating synthetic text-to-SQL samples. They also introduce a Mixture of Experts approach to merge dialect-specific SQL models into a single model for efficient cross-dialect knowledge sharing and reducing maintenance costs by selectively activating dialect-specific experts based on the input. The method is tested on BIRD and Paglia benchmarks and is evaluated against baseline datasets to validate SQL-GEN's efficacy.

strengths:
- **Originality**
    - The task of addressing dialects in text-to-SQL systems is nascently explored, and this paper makes a valuable contribution by introducing a novel MoE approach to manage multi-dialect scenarios effectively.
- **Quality and Clarity**: 
     - The paper is generally easy to follow.
     - The experiments comprehensively test SQL-GEN across multiple benchmarks and datasets and validate the performance.

weaknesses:
- Figures, especially Figures 3 and 4, are somewhat dense and challenging to interpret. A clearer visual representation or more detailed explanations would enhance accessibility.
- The text in Figure 5 could be enlarged to improve readability, as it currently appears difficult to see.

questions:
- On a side note, it would be interesting to consider cross-lingual transfer methods for text-to-SQL as seen in multilingual NLP tasks, which might provide a useful baseline for comparison.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698666021

thread_type:
official_review

<thread_object>
creator: reviewer_gbqt

<content>
summary:
This paper introduces the SQL-GEN framework, which bridges the dialect gap for text-to-SQL systems by generating high-quality synthetic data for any dialect and merging dialect-specific models into a unified model using a novel Mixture of Experts (MoE) initialization method. The framework significantly improves execution accuracy on unseen real-world multi-dialect benchmarks and reduces the gap compared to large-scale human-annotated data.

strengths:
1. The method is evaluated on three SQL dialects and proves its effectiveness.
2. The author has built a text-to-sql dataset that includes SQL dialects. If this dataset is open source, it will be of certain value to research.

weaknesses:
1. The paper presents contributions that, while valuable, appear to have limited novelty compared to existing work in the field[1].
2. In addition, from the results, the improvement of synthetic data is not as good as that of bird train set.

[1] Yu T, Wu C S, Lin X V, et al. Grappa: Grammar-augmented pre-training for table semantic parsing[J]. arXiv preprint arXiv:2009.13845, 2020.

questions:
No more questions. Please see above.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a method for generating synthetic data for text-to-SQL tasks in multiple "dialects" (e.g., BigQuery, PostgreSQL, SQLite). This is done by generating abstract queries which are then refined into each dialect.  Training on this data produces effective models for each setting. These models are then merged by initializing an MoE model with each expert using a novel procedure. Results across several benchmarks compare training on the synthetic data from this work to training on existing datasets such as BIRD, as well as the performance of the MoE model versus individual expert models.

This paper presents its problem setting very clearly, proposes a clear solution, and evaluates that solution very thoroughly. The mixture-of-experts approach is an interesting way to combine models.

The main weaknesses of the work are the model performance. Comparing to training on this paper's synthetic data to training on BIRD, and comparing MoE and SFT, the results are mixed (though generally positive in favor of this paper). The authors assert that their approach provides some conceptual advantages in both settings (e.g., generality compared to BIRD-trained models, where in-domain test sets are artificially easy). However, it's not clear to me whether the needle has been substantially moved in this domain (i.e., whether a broad range of practitioners in text-to-SQL really need to pay attention to these results), and also whether there's a generalizable takeaway method for a typical attendee of ICLR.

Overall, this paper delivers data and systems that will be useful to practitioners in this area, but I'm not sure these will be as useful or interesting to the broader ICLR audience.