Introduction
Text-to-SQL systems aim to convert natural language questions into SQL queries that can be used to query a database. The systems serve as an interface between users and databases to allow the users access to information from the databases through their natural language questions. The advent of Large Language Models (LLMs) Bubeck et al. ( 2023 ) has significantly enhanced the capabilities of text-to-SQL systems, such as DIN-SQL Pourreza and Rafiei ( 2024 ) , achieving state-of-the-art (SoTA) performance on standard benchmarks 2 2 2 As of August 2023 , including Spider Yu et al. ( 2018 ) and BIRD Li et al. ( 2024 ) .

Although the SoTA text-to-SQL systems perform well on clean benchmarks that contain only answerable user queries, they are still not well-equipped to deal with practical real-world data which have ambiguous or unanswerable questions Wang et al. ( 2023a ) . The poor performance of SoTA text-to-SQL systems is primarily due to the unavailability of practical text-to-SQL data that can be used for training Wang et al. ( 2023a ) . Although previous research finds that a large ratio of user questions are unanswerable, these are often excluded in the previous datasets as addressing them requires more than SQL annotations Lee et al. ( 2021 ) .
To bridge this gap, we introduce PRACTIQ which is a practi cal conversational text-to-SQL dataset with ambiguous and unanswerable q ueries. As illustrated in Table 2 , a question is ambiguous if it has multiple valid interpretations given the database schema and the question is unanswerable if the corresponding database does not contain the data that the question is asking for. In the real world, given a user question, a text-to-SQL assistant has to first determine whether the question is answerable, ambiguous, or unanswerable to decide whether to ask for clarification questions or respond with the correct SQL.

We begin by examining existing text-to-SQL datasets Yu et al. ( 2018 ); Li et al. ( 2024 ); Yu et al. ( 2019a ) and identify four ambiguous and four unanswerable categories inspired by real-world practical user questions. Subsequently, we generate ambiguous and unanswerable examples corresponding to these categories by parsing the SQLs and modifying the databases (Spider 3 3 3 We only used “train_spider.json” and “dev.json” from Spider in our dataset construction and related experiments. We did NOT use any data from “train_others.json” or “yelp”. is used in the current work, but the framework can be easily adapted to other text-to-SQL datasets). We then leverage an LLM to convert the data into conversations between the user and a text-to-SQL assistant that includes user initial questions, assistant clarification questions, user clarification responses, assistant SQL responses, SQL execution results, and natural language explanations of the execution results (as shown in Figure 1 ). In addition to having conversations where the assistant asks for clarification questions, we also generated more helpful SQL responses that included the results of all possible responses for some ambiguous question categories. To assess the quality of our generated dataset we define annotation criteria for two tasks, question category classification, and conversation quality evaluation, and conduct human annotation on the generated data to show that our dataset is of high quality. Finally, we propose prompt-based baselines to benchmark our dataset on the text-to-SQL generation task, which involves two tasks, classifying the category of the user question, and then generating the clarification SQL based on the user question. We experiment with several SoTA LLMs and show that the current text-to-SQL systems still need improvements on real-world queries that include ambiguous or unanswerable questions.

Our contributions can be summarized as follows:

We study existing text-to-SQL datasets and identify four ambiguous and four unanswerable question categories inspired by real-world user questions. We implemented a framework and programmatically generated PRACTIQ , a comprehensive and fine-grained ambiguous and unanswerable text-to-SQL dataset consisting of 2800 conversations.

We extend the ambiguous/unanswerable data into conversations between a user and an assistant. The conversation typically includes a user initial question, a helpful assistant response seeking user clarification, a user clarification response, the assistant SQL response, SQL execution results, and a natural language explanation.

To the best of our knowledge, our work is the first to study text-to-SQL systems when user queries can be answerable, ambiguous, or unanswerable in a conversational setting.
We benchmark various SoTA LLMs on PRACTIQ on two sub-tasks:
question category classification, and clarification SQL prediction. Our results show that the ambiguous and unanswerable questions are challenging even for methods leveraging SoTA LLMs indicating the need to improve LLMs’ handling of real-world practical text-to-SQL data.