MultiSQL: A Schema-Integrated Context-Dependent Text2SQL Dataset with Diverse SQL Operations (2024)

Abstract: Text2SQL is a task that translates natural language into SQL statements. Context-dependent Text2SQL offers a more natural database interaction by simulating dialogues between users and databases, with CoSQL and SparC as representative datasets. Yet, these datasets struggle to accurately replicate real-world situations. To address this, we introduce MultiSQL, which extends them in three key aspects: (1) Diverse SQL Operations. We incorporate diverse SQL types such as Create, Update, and Insert to broaden the scope of SQL operations. (2) Schema-Integrated Context. We integrated query context with database schema dependencies to better depict database complexity. (3) Extended Dialogues. We expand dialogue length to better simulate long conversations and complex interactions. This multitype, schema-integrated, context-dependent Text2SQL dataset comprises nearly 800 dialogue groups and over 9,000 interaction turns across 166 complex databases, offering a better benchmark for interactive user-database dialogue. Addressing MultiSQL’s challenges, we refined evaluation metrics to better capture diverse SQL types and schema dependencies. We designed a prompt framework that leverages historical data and self-refinement to accurately capture the dependency between text queries and database structures. Experiments with GPT-3.5, GPT-4, and LLaMA27B show both the effectiveness of our strategies and the challenges of MultiSQL. The datasets is available at https://github. com/grandchicken/MultiSQL.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 13857–13867 August 11-16, 2024 ©2024 Association for Computational Linguistics
Text2SQL is a task that translates natural language into SQL statements. Context-dependent Text2SQL offers a more natural database interaction by simulating dialogues between users and databases, with CoSQL and SparC as representative datasets. Yet, these datasets struggle to accurately replicate real-world situations. To address this, we introduce MultiSQL, which extends them in three key aspects: (1) Diverse SQL Operations. We incorporate diverse SQL types such as Create, Update, and Insert to broaden the scope of SQL operations. (2) Schema-Integrated Context. We integrated query context with database schema dependencies to better depict database complexity. (3) Extended Dialogues. We expand dialogue length to better simulate long conversations and complex interactions. This multitype, schema-integrated, context-dependent Text2SQL dataset comprises nearly 800 dialogue groups and over 9,000 interaction turns across 166 complex databases, offering a better benchmark for interactive user-database dialogue. Addressing MultiSQL’s challenges, we refined evaluation metrics to better capture diverse SQL types and schema dependencies. We designed a prompt framework that leverages historical data and self-refinement to accurately capture the dependency between text queries and database structures. Experiments with GPT-3.5, GPT-4, and LLaMA27B show both the effectiveness of our strategies and the challenges of MultiSQL. The datasets is available at https://github. com/grandchicken/MultiSQL.

heading: 1 Introduction
text: In the information age, structured data is predominantly stored in databases. We interact with them through SQL, a query language specifically designed for managing and manipulating databases. However, SQL is quite complex, requiring a deep
∗ Corresponding author.
understanding of database structures and query syntax for effective data retrieval. To address this, the Text2SQL (Katsogiannis-Meimarakis and Koutrika, 2023; Qin et al., 2022; Dong et al., 2023; Pourreza and Rafiei, 2023; Gao et al., 2023) task was proposed, allowing direct conversion of natural language into SQL queries. This enables users to interact with databases directly using natural language, greatly facilitating data access.
To assess the effectiveness of Text2SQL, several datasets have been developed. Spider (Yu et al., 2018) is the first proposed dataset in a crossdatabase context. It includes complex SQL types like groups, joins, and nested queries, effectively measuring model adaptability for this task. Bird (Li et al., 2023) goes further towards real-world application, incorporating noisy data and external knowledge, to accurately depict the complexity of data in real-world business scenarios, thus challenging
13857
models in processing real-world data. However, the above datasets assume SQL queries are stated in a single sentence, overlooking the reality that Text2SQL interactions typically occur as dialogues. In real-world scenarios, it’s a more natural way for Text2SQL applications to involve users dynamically interacting with databases through dialogue. Therefore, context-dependent Text2SQL has been developed, with SparC (Yu et al., 2019b) and CoSQL (Yu et al., 2019a) serving as pivotal datasets for this task. SparC simulates the user-database interactive dialogues, while CoSQL is a corpus developed for building cross-domain, general-purpose database querying dialogue systems. In the context-dependent Text2SQL task, models must understand the context to create the right SQL statements for the current queries.
However, existing context-dependent Text2SQL datasets still have limitations in accurately replicating real-world situations. To address this, we have extended them in the following three aspects:
1. Diverse SQL Operations: Previous datasets were limited to Select queries. In real situations, users engage with databases not just for queries but also for modifying and managing database structures and content. Therefore, we have incorporated SQL statement types such as Create, Update, Insert, Alter, and Delete, making our dataset more comprehensively reflect database interactions.
2. Schema-Integrated Context: Existing contextdependent Text2SQL datasets focused solely on natural language context dependencies. However, dependencies also exist within the database schema context. For instance, as shown in Figure 1, a column added by the user through an Alter Table in earlier dialogue may change the structure of table schema, which is essential when it is queried later by a Select statement. This requires the interaction system to dynamically capture changes in the database structure during the conversation. To address this, we have integrated the dependencies between database table structures into our dataset, more accurately simulating the complexity and dynamics of real-world databases.
3. Extended Dialogues: According to statistics, existing datasets have a relatively low average number of dialogue turns. In our new
dataset, we have greatly increased the number of dialogue interactions to better simulate long conversations and complex interactions in real-world scenarios.
Consequently, we propose a Multi-type, schemaintegrated, and context-dependent Text2SQL dataset, called MultiSQL covering various types of SQL operations. It comprises nearly 800 dialogue groups, with over 9,000 turns spanning 166 complex databases. Table 1 shows its advantages compared to existing datasets.
Our dataset brings new challenges to method design and evaluation system construction. For methods, the integrated context necessitates that prediction processes thoroughly consider the semantic context generated during dialogue and changes in database structure. The presence of diverse SQL types, along with longer dialogue histories, makes it easier for models to accumulate errors during interactions. To address this, we’ve developed a prompt framework, integrating historical data and a self-refinement (Peng et al., 2023) mechanism to mitigate prediction errors.
For evaluation, traditional metrics include Exact Set Match and Execution Accuracy. However, for MultiSQL, the former does not account for structural changes, and the latter struggles to handle issues with non-select statements that do not return values. Consequently, we introduce Context-Aware Match to track dialogue-induced table structure changes, and Database State Match to assess SQL execution effectiveness by comparing changes of the database content.
Finally, our experiments on models like GPT3.5 (Ouyang et al., 2022), GPT-4 (Achiam et al., 2023), and LLaMA2-7B (Touvron et al., 2023) highlight the difficulties inherent in the MultiSQL dataset and the effectiveness of our proposed approaches.
Our contributions are summarized as follows:
1. We present MultiSQL, a Multi-type, schemaintegrated, and context-dependent Text2SQL dataset. By incorporating different types of SQL operations and integrating database schema dependencies in a long dialogue, we make our dataset a more comprehensive reflection of the complexity and dynamism in real-world database interactions.
2. In response to the challenges presented by our
dataset, we have accordingly developed methods and introduced new evaluation metrics.
3. Our experiments on models like GPT-3.5, GPT-4, and LLaMA2-7B effectively show the challenges posed by the MultiSQL dataset and the efficacy of our methods.

heading: 2 Related Work
text: Text-to-SQL aims to automatically translate natural language questions into SQL queries. The development of text-to-SQL datasets has evolved to more closely resemble real-world scenarios, reflecting an increase in complexity and a move towards more accurately simulating real-world data interactions.
Initially, datasets in this field were relatively simple and focused on single-domain scenarios. Early datasets such as GeoQuery (Zelle and Mooney, 1996), ATIS (Price, 1990), and Restaurant (Dahl et al., 1994)targeted specific information retrieval tasks within a limited domain. These datasets laid the groundwork for future advancements but were limited in their scope and complexity.
Considering that Text2SQL in real scenarios is an open-domain problem, the field has seen a shift towards cross-domain datasets, exemplified by
WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018). These datasets broadened the scope from single-domain to cross-domain, requiring models to generalize across various domains. However, a common limitation in many cross-domain datasets is their focus on the database schema without adequately considering the specific values within the tables, diverging from the complexity of real-world databases.
To address this, datasets like KaggleDBQA (Lee et al., 2021), EHRSQL (Lee et al., 2022), SEDE (Hazoom et al., 2021), and MIMICSQL (Wang et al., 2020) were introduced. These datasets focused on large-value databases and professional SQL queries, thereby moving closer to real-world database applications. Bird further advanced this trend by incorporating noisy data and external knowledge into its structure to accurately depict the complexity of data in real-world business scenarios, making it more challenging and representative of real-world data processing scenarios.
Another significant development in Text-to-SQL datasets is the recognition that it’s more natural for Text2SQL applications to involve users dynamically interacting with databases through di-
alogue. Therefore, context-dependent Text2SQL has been developed. SparC simulates user-database interactive dialogues, requiring models to understand and respond to context-dependent queries. CoSQL, on the other hand, is designed for building cross-domain, general-purpose database querying dialogue systems. It challenges systems in SQLgrounded dialogue state tracking, response generation, and user dialogue act prediction, closely mimicking real-world dialogue scenarios.

heading: 3 Dataset Construction
text: Due to CoSQL being a representative example of a context-dependent Text2SQL dataset, we adopted its database in our work. Building upon this foundation, we constructed a Text2SQL interaction dataset to further explore and address the complexities of natural language interfaces to databases. Due to the high cost and extensive time requirements associated with manual data annotations, we use large language models as an aid in the dataset construction process like much recent research does (Akyürek et al., 2023; Chen et al., 2023; Zhang et al., 2023b). We designed a set of prompt frameworks and used GPT-4 to generate data. Subsequently, we manually corrected and modified the generated data, and iteratively checked and improved the quality of the dataset. Below, we will provide a detailed description of the entire dataset construction process.

heading: 3.1 Prompt Design
text: Our prompt design, depicted in Figure 2, integrates several key components: (1) System Information, providing a task overview, dataset attributes, and input-output formats; (2) Table Schema, detailing the database structure including table and field names, data types, and keys; (3) Table Content, presenting the initial entries of each table and the total count of entries; (4) Demonstrations, offering initial Select queries from CoSQL to lay the groundwork for the model’s understanding of database queries; and (5) Chain of Thought, employing a reasoning method to guide the model in formulating queries with accurate logic.

heading: 3.2 Data Generation
text: Here, we employed GPT-4 to generate data based on the above-mentioned prompt framework. Adding more detail to the situation, we generated 817 sets of dialogues, comprising a total of 9317 pairs of query-SQL data. Afterward, by filtering
out the duplicates, we ultimately obtained 783 sets of dialogues, with 8923 pairs of query-SQL data.

heading: 3.3 Manual Correction and Improvement
text: However, the generation process by GPT-4 can only serve as an auxiliary tool. The data produced by GPT-4 still has some issues. Therefore, we invested substantial human effort to refine the base data generated by GPT-4, aiming to correct existing issues and enhance the overall quality of the dataset. This involved several key steps:
• Grammar Correction: We validated and corrected grammatical errors in 12% of the generated SQL statements to ensure adherence to SQL syntax standards. This included resolving 6% of errors due to conflicts in insert statement numbering from inability to accurately retrieve entry counts in the database, 4% for non-compliance with SQL syntax rules, and 2% for inserting duplicate fields.
• Semantic Correction: We identified and rectified semantic misalignments in 4% of the data, such as mapping queries to the incorrect tables, failing to understand temporal information in questions, and inferring missing context from the queries inaccurately
• Contextual Relevance Improvement: We inserted 334 query-SQL interactions based on dialogue context to enhance the relevance of semantic association within the dialogue context and the dependency on table schema, addressing the complexities of real-world database interactions.
Building on the previously mentioned steps, our team dedicated 114 person-hours to the postprocessing, correction, and improvement of the generated data. This substantial effort in grammar correction, semantic accuracy, and contextual relevance was instrumental in ensuring the high quality of the dataset.

heading: 4 Data statistics and analysis
text: Table 2 presents the statistical information of our dataset and compares it with existing datasets. It reveals that our dataset comprises a total of 9257 query turns, covering 166 databases. This scale of query turns is comparable to mainstream datasets.
Moreover, it’s notable that our dataset has an average of 11.81 query turns per dialogue group, which is significantly higher than CoSQL and SParC.
Figure 3 shows the distribution of SQL types in our dataset, excluding Delete statements which are rare in practice. The balance among other operations is notable, with Insert at 25.83%, Select at 25.8%, and UPDATE at 23.19%. This reflects the varied use of SQL in real situations, enhancing our dataset’s relevance and providing a broad testing ground for systems handling diverse SQL queries within dialogues.

heading: 5 Method
text: MultiSQL, with Diverse SQL Operations, SchemaIntegrated Context, and Extended Dialogues, presents unique challenges to the method design.
First, the challenge arises from the need to understand the semantic context within dialogues and adapt to database schema changes. To address this, we integrate historical data, a strategy that involves incorporating past interactions into the model’s current decision-making process. This approach ensures that the model not only grasps the immediate query but also contextualizes it within the dialogue’s history, enhancing its ability to adapt to and reflect changes in the database structure ac-
curately. By doing so, we mitigate issues related to semantic understanding and database adaptation, ensuring more accurate and contextually relevant predictions.
Second, the diversity in SQL types and the potential for error accumulation through extended dialogues require a robust mechanism to maintain model accuracy. Here, our solution is the implementation of a self-refinement mechanism. This process involves continuously analyzing model predictions for errors and refining the model’s strategies based on feedback. Such a mechanism directly tackles the accumulation of errors by enabling the model to learn from its mistakes and adjust its approach for future queries, thus enhancing its reliability and accuracy in handling a wide range of SQL operations within prolonged dialogues.
Thus, we have devised a prompt framework that integrates historical data and incorporates a selfrefinement mechanism. The framework contains several parts as shown in Figure 4:
• System Initialization: Starts with a prompt that outlines the task, database schema, and initial data context.
• Integrating Dialogue History: Adds previous dialogue excerpts to the current query for context.
• Self-Refinement Mechanism: Uses an SQL executor to validate and provide feedback on the model-generated SQL for correction and improvement.
In this way, our approach systematically addresses the challenges posed by MultiSQL through initial setup, context integration, and iterative refinement. Section 7 experiments further validate the effectiveness of our methodology.

heading: 6 Evaluation Metrics
text: In Text2SQL tasks, traditional evaluation metrics include Exact Set Match and Execution Accuracy.
• Exact Set Match: This metric assesses the equivalence of the predicted query to the gold query across specific SQL components: FROM, WHERE, GROUP BY, etc. Each component is evaluated for an exact match between the predicted and gold queries. The predicted query is deemed correct if, and only if, all components match exactly with their counterparts in the gold query.
• Execution Accuracy: This metric executes both the predicted and gold SQL queries on the database and compares the result sets. If the result sets are identical, the queries are considered equivalent.
However, these metrics encounter specific challenges when applied to our dataset.
• Challenges with Exact Set Match: First, Create and Alter statements can lead to naming variations in fields, as depicted in Figure 5(a), where different but valid table names reflect the same query’s intent. Second, as table structures change during dialogues (Figure 5(b)), accurately matching subsequent queries necessitates considering previous modifications
• Challenges with Execution Accuracy: Beyond Select statements, other types of SQL queries do not produce a direct return value, making it challenging to evaluate their execution effectiveness.
Consequently, to address the distinctive challenges of our dataset, we introduce two novel evaluation metrics: Context-aware Match which gauges
the alignment of predicted and actual table structures within dialogue contexts, and Database State Match, which measures SQL effectiveness by examining database state alterations. Further details are provided in the following subsections.

heading: 6.1 Context-aware Match
text: To tackle the issue of field naming discrepancies, we first introduce fuzzy matching which aims to accommodate variations in naming conventions by allowing for a more flexible comparison between predicted and actual field names. Fuzzy matching operates under the principle that a match is recognized if:
1. The field name in the answer (s) is contained within the predicted field name (c), or vice versa;
2. Parts of s split by underscores (_) are found within c, or parts of c split by underscores are found within s.
Additionally, for operations like Create and Alter that change table structures, we track structural modifications by recording added (Tadd and Cadd), deleted (Tdel and Cdel), and altered (Calt) elements. This leads to the creation of a mapping dictionary (M ) that correlates predicted and actual database elements. During SQL evaluation, a context-aware match score of 1 is awarded for exact field matches as per Mcolumn, with any deviation resulting in a 0.
Score =    1 if named fields match according to Mcolumn and all non-named fields align perfectly
0 otherwise

heading: 6.2 Database State Match
text: For evaluating non-Select statements, we refine our approach with the Database State Match metric. This metric contrasts the database states after executing the predicted SQL statement and the reference SQL statement, aiming to verify if these resulting states are identical. This comparison is essential for operations such as Create, Alter, and Delete, which impact the database’s structure, and for Insert and Update, which alter its content.
The matching score K is thus recalculated to reflect the alignment between the predicted and actual outcomes on the database. The formula is adjusted as follows:
K =    Structural Match(Dpred, Dgold), for Create, Alter, Delete
Content Match(Dpred, Dgold), for Insert, Update
In this context, Dpred and Dgold represent the database states after executing the predicted and gold standard SQL statements respectively. The Structural Match verifies the structural integrity and schema modifications aligned between Dpred and Dgold. while the Content Match ensures an exact content match in the database following the execution of both predicted and gold SQL statements.

heading: 7 Experiments

heading: 7.1 Experimental Setup
text: In our experiments, we employed GPT3.5 (Ouyang et al., 2022), GPT-4 (Achiam et al.,
2023), and LLAMA2-7B (Touvron et al., 2023). Additionally, we introduced domain-specific models for the code domain: TableLlama-7B (Zhang et al., 2023a), CodeLlama-7B (Roziere et al., 2023), and Deepseeker-Code-7B (Guo et al., 2024). We used a prompt framework and in-context learning to interact with these Large Language Models, maximizing their response capabilities to structured prompts. Specifically, for LLAMA2-7B, we adopted two approaches: a zero-shot setting where prompts were directly inputted for inference, and an instruction-tuned setting, where we utilized 662 groups from our dataset to construct 8006 instruction-tuning pairs. The remaining 121 groups were then employed for testing. For more implementation details, see Appendix A.
For prompt configurations, we experimented with several settings: (1) Baseline, which processes text directly to generate SQL statements without additional context; (2) History, which enhances inputs by appending historical dialogue data before SQL generation; and (3) Self Refine, which introduces a feedback loop from executing generated SQL to refine subsequent outputs.
Our main evaluation metrics are as follows: (1) Execution Accuracy, which focuses on the precision of executing SQL select statements; (2) Context-aware Match, adopting the previously mentioned strategy and assessing accuracy across all types of statements; and (3) Database State Match, evaluating the congruence of the database state post-execution, applicable to all statement types except select statements.

heading: 7.2 Experimental Results
text: Table 3 offers insightful results regarding the performance of different models and methods across key metrics such as Context-aware Match, Execution Accuracy, and Database State Match. In comparing models GPT-3.5, GPT-4, and LLAMA2-7B, the Self Refine method generally outperforms the Baseline and History methods across the board. For Context-aware Match, Self Refine achieves top scores in the Create and Update categories, with GPT-4 reaching 0.345 and 0.786 respectively. The Execution Accuracy for Select is also highest with Self Refine, scoring 0.701 for GPT-4 and 0.653 for GPT-3.5. Database State Match scores indicate Self Refine leads in Insert, Create, and Update actions, with GPT-4 scoring 0.586, 0.529, and 0.735 respectively.
LLAMA2-7B, even when tuned, shows a stark contrast in performance compared to GPT models. The tuned LLAMA2-7B’s best Context-aware Match scores after Self Refine are 0.059 for Select and 0.155 for Update, a considerable gap from GPT-4’s performance. This highlights the LLAMA2-7B’s limitations in complex SQL tasks and underscores the challenging nature of the dataset which demands robust contextual understanding and adaptability from models. The clear disparity in the results illustrates the importance of methodological refinement in achieving high accuracy on this demanding dataset.
Table 4 and Table 5 offer insightful results regarding the performance of different models and methods across key metrics such as Contextaware Match, Execution Accuracy, and Database State Match. These tables compare the models TableLlama-7B, CodeLlama-7B, and DeepseekerCode-7B using Baseline, History, and Self Refine methodologies.
We can find that Deepseek-Coder-7B consistently performs the best, demonstrating its robust capability in handling various types of Text2SQL tasks. In contrast, CodeLLAMA-7B and TableLLAMA-7B struggle to match this performance, highlighting the challenging nature of our dataset. For the less capable models, CodeLLAMA-7B and TableLLAMA-7B, sophisticated prompt methods like History and Self Refine do not always yield the best results. This may reflect their weaker instruction comprehension capabilities. However, for Deepseek-Coder-7B, and the previously discussed GPT-3.5 and GPT-4, the benefits of the Self Refine method become increasingly apparent as their instruction comprehension
abilities improve. This trend indicates that the performance of these models can be significantly enhanced with better contextual understanding and methodical refinement.

heading: 7.3 Detailed Analysis
text: We provide a detailed assessment of the Create and Insert SQL statements for models GPT-3.5, GPT-4, and LLAMA2-7B, with a focus on Table Name Accuracy, which is the correct prediction rate of table names, and Field Match Ratio, which is the precision of field name prediction. These detailed metrics are designed to finely gauge the models’ predictive capabilities regarding database structure.
The experimental outcomes for GPT-3.5 and
GPT-4 demonstrate their proficiency. GPT-4 with the Self Refine method reaches near-perfect Table Name Accuracy scores of 0.984 for Create and 0.988 for Insert, and Field Match Ratio scores of 0.769 for Create and 0.963 for Insert. These findings robustly validate the effectiveness of the Self Refine, especially in predicting table names, where it exhibits an almost flawless performance.
In contrast, LLAMA2-7B’s performance paints an interesting picture. The model, without any fine-tuning, records substantial Field Match Ratio scores of 0.464 for Table Name Accuracy in Create and 0.458 in Insert. These figures stand out against the backdrop of nearly zero scores in the primary experimental metrics of Context-aware Match and Database State Match. This indicates that LLAMA2-7B possesses a partial ability to predict Insert statements and shows a preference for generating table names and some database fields accurately. However, after fine-tuning, there is a noticeable decrease in performance, possibly because such tuning leads the model to approach Insert statement prediction from a more global perspective. This change might undermine its earlier predictive abilities, pointing to a nuanced trade-off between general and detailed SQL skills influenced by fine-tuning.

heading: 7.4 Case Study
text: In our case study on GPT-4 in Figure 6, we compared Baseline, History, and Self Refine methods using the same query. The Baseline method fails to deduce the correct table and column names due to lack of context. The History method corrects the table name but uses NULL for the ID, violating the non-null primary key constraint. The Self Refine
method successfully rectifies the table name and uses a subquery to compute the next ID, generating the correct SQL statement. This study highlights that the History method improves context perception over Baseline, while Self Refine excels by using database feedback to correct errors. Incorporating such feedback is crucial for generating contextually accurate and constraint-respecting SQL queries.

heading: 8 Conclusion
text: In conclusion, we introduce MultiSQL, a Multitype, schema-integrated, and context-dependent Text2SQL dataset, designed to closely mirror the complexities and dynamism of real-world database interactions. By incorporating a diverse range of SQL operations and embedding database schema dependencies within extended dialogue interactions, MultiSQL offers a significantly more nuanced and challenging environment for Text2SQL applications.
Limitations
The MultiSQL dataset, while advancing the Text2SQL domain, encounters limitations in fully
replicating the complexity of real-world database scenarios, potentially affecting its generalizability. With 166 databases, the variety, although extensive, may not encompass the vast diversity of real-world database schemas, limiting the dataset’s applicability across different domains. Additionally, the refined evaluation metrics, though improved, might not capture all aspects of SQL query quality such as runtime efficiency and adherence to SQL writing best practices. This could lead to a gap in measuring the true effectiveness of SQL queries generated from natural language interactions, highlighting areas for future enhancement to bridge the gap between simulated environments and real-world database usage.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments. This work is supported by the National Natural Science Foundation of China (No. 62206126, 61936012 and 61976114).

heading: Future Work
text: This could lead to a gap in measuring the true effectiveness of SQL queries generated from natural language interactions, highlighting areas for future enhancement to bridge the gap between simulated environments and real-world database usage