RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
B.2 Conversation Quality Evaluation
We sampled 90 conversations across ambiguous, unanswerable, and answerable categories from different databases for human annotation. Two SQL experts annotated each conversation on three criteria: factuality (correctness of SQL and natural language response), helpfulness (assistant’s responses in understanding user intent), and naturalness (conversation flow) using a 1-5 Likert scale, where 1 denotes perfect/best quality and 5 denotes the worst quality.

Table 4 shows the mean, standard deviation, and Krippendorff’s Alpha for inter-annotator agreement. The high mean scores close to 1 (1.15-1.5) and substantial agreement (Alpha 0.68-0.82) indicate high-quality, natural conversations with factual and helpful responses. For Naturalness, we observe that annotators have a substantial agreement (Krippendorff’s Alpha = 0.82), indicating that the conversations are generally perceived as natural and fluent. For Factuality, the annotators demonstrate moderate agreement (Krippendorff’s Alpha = 0.68), suggesting that the conversations are consistently viewed as highly factual, which implies that the SQL queries in our dataset are of high quality. For Helpfulness, the annotators show good agreement (Krippendorff’s Alpha = 0.76), indicating that the conversations are mostly helpful. Overall,

Table 8 presents category-wise annotation scores. Answerable data from Spider has a mean of 1 across criteria, confirming its high quality. Naturalness scores closer to 2 for most categories indicate mostly natural conversations, with 1 being perfectly natural. Categories like Ambiguous Filter Criteria, Nonexistent Filter Value, and Nonexistent Where Column have the most natural conversations (mean scores closer to 1), likely due to the close relation between user follow-up and assistant responses.

Factuality scores close to 1 across categories demonstrate accurate SQL generation and result descriptions through our reverse generation process, with 1 being perfectly factual. Helpfulness scores around 1.5 suggest mostly helpful assistant responses in understanding user intent, with 1 being perfectly helpful. Higher standard deviations for certain categories (e.g., Ambiguous Where Column, Ambiguous Values Within Column) indicate annotator disagreements due to varying relevance of ambiguous interpretations to user queries across examples.

Overall, the human annotation results validate the high quality, naturalness, factuality, and helpfulness of the generated conversational data.
E.1 Ambiguous WHERE Column
In stage 1, we collect the column names appearing in the Where clause of the SQLs of all questions in the Spider dataset. Like in the Ambiguous SELECT Column case, we then provide those columns as inputs to an LLM and prompt it to generate column names that are semantically similar but not equivalent. We then construct the value of ambiguous problematic data by removing the original column from the schema and adding the newly generated Ambiguous SELECT Columns. In stage 2, we construct the assistant’s helpful response using a template similar to the Ambiguous SELECT Column case. We generate the assistant’s clarified SQL by replacing the column in the Where clause of the original SQL with one of the Ambiguous SELECT Columns. We then provide this data in the form of a conversation as input to the LLM and prompt it to come up with the user clarification response. Like in the Ambiguous SELECT Column case, after generating the user clarification we filter the data based on some rules. We discard those samples that contain the new column or miss the removed column in the clarification. We finally refine the conversation, execute the clarification SQL to get the results, and generate the natural language explanation of the results.
E.2 Ambiguous Values Within Column
In stage 1, we extract the values appearing in the Where clause of the SQLs of all questions in the Spider dataset. We prompt the LLM to generate values that are similar but not equivalent to each other. We then construct the Ambiguous Values Within Column ambiguous data by constructing a new schema from the original schema by removing the original value and adding the newly generated ambiguous values. For example, for the value “chemistry” the LLM generates two ambiguous values, “organic chemistry” and “physical chemistry”. In stage 2, we construct the assistant’s helpful response by using a template that points out the presence of two Ambiguous SELECT Columns. We generate the assistant’s clarified SQL by replacing the original value with one of the ambiguous values. We then provide this data in the form of a conversation as input to the LLM and prompt it to come up with the user clarification response. We then discard those data where the user clarification does not mention the newly generated values. Finally, we refine the conversation, execute the clarification SQL get the results, and generate the natural language explanation of the results.
E.3 Ambiguous Filter Criteria
To construct the ambiguous filter criteria data, we utilized the SPIDER dataset. Instead of modifying the databases, we prompted a Large Language Model (LLM) to modify the user questions to introduce ambiguity. Specifically, we employed the following techniques:
1. Replacing specific filter values with relative terms like "little/large," "young/old," "slow/fast," etc.
2. Using descriptive terms instead of explicitly stating the original filter value.
The modified questions resembled those in the BIRD dataset that require additional "evidence" or definitions to convert text to SQL. After modifying the questions, we prompted the LLM with different instructions to generate a response from the database assistant’s perspective, explaining why the question was ambiguous.
Finally, we used the original (unmodified) user question as the clarified follow-up question, and the corresponding SQL as the gold SQL after the user’s clarification.
E.4 Nonexistent SELECT Column
In stage 1, we extract the columns appearing in the Select clause of the SQLs of all questions in the Spider dataset and construct new schemas by removing the columns required for answering the respective questions. In stage 2, we construct the assistant’s helpful response using a template that states that the column required for answering the question is missing from the schema. We construct the final SQL by replacing the missing column from the schema (in the Select clause) with a column that exists in the schema. We convert this data into conversational data and prompt the model to generate the user clarification response. In stage 3, we refine the conversation, execute the clarification SQL get the results, and generate the natural language explanation of the results.
E.5 Nonexistent Filter Value
In stage 1, we extract the values appearing in the Where clause of the SQLs of all questions in the Spider dataset. For constructing the problematic data we construct a new schema by removing the values required for answering the question from the schema. In stage 2, we construct the assistant’s helpful response using a template that mentions that the value mentioned in the question is not present in the schema. We construct the clarification SQL by replacing the removed value with another value present in the schema. We then convert this data into conversational data and prompt the LLM to generate the user clarification response. In stage 3, we refine the conversation, execute the clarification SQL get the results, and generate the natural language explanation of the results.
E.6 Unsupported Join
In stage 1, to construct the problematic data, we consider the unique schemas of the Spider dataset and prompt the LLM to generate a new schema with at least two new tables and corresponding columns such that the new tables have a foreign key relationship with themselves but not with any other column in the schema. For example, for a schema containing student information like student grade, teacher details, etc. the LLM produces two new tables of library and books that have a foreign key relationship with each other but not with any other table in the original schema. In stage 2, we construct the assistant’s helpful response using a template stating that the question requires joining tables of the schema that have no relationship with each other. We construct a clarification SQL by using SQL from the Spider dataset corresponding to the original schema. We then convert this data into conversational data and prompt the LLM to generate the user clarification response. In stage 3, we refine the conversation, execute the clarification SQL get the results, and generate the natural language explanation of the results.
E.7 Nonexistent WHERE Column
In stage 1, we extract the columns present in the Where clause of the SQLs in the Spider dataset and construct new schemas by removing the columns required for answering the respective questions. In stage 2, we construct the assistant’s helpful response using a template that mentions that the information required for answering the question is not present in the schema. We construct the clarification SQL by finding a SQL from the Spider dataset whose Select columns match the problematic question and whose Where columns are present in the schema. We convert this data into a conversational format and prompt the LLM to generate the user clarification response. In stage 3, we refine the conversation, execute the clarification SQL get the results, and generate the natural language explanation of the results.
Figure 1: An example of our conversational dataset construction consists of three stages: database modification, SQL modification along with clarification response generation, and refining the conversation. The top box depicts our data construction for an ambiguous question that requires clarification questions, while the bottom box illustrates an ambiguous question with direct helpful SQL responses. Here ‘U’ refers to a user and ‘A’ refers to a text-to-SQL assistant.; An example of our conversational dataset construction consists of three stages: database modification, SQL modification along with clarification response generation, and refining the conversation. The top box depicts our data construction for an ambiguous question that requires clarification questions, while the bottom box illustrates an ambiguous question with direct helpful SQL responses. Here ‘U’ refers to a user and ‘A’ refers to a text-to-SQL assistant.
Figure 2: Figure showing the classification accuracy of different models using different number of shots.; Figure showing the classification accuracy of different models using different number of shots.
Figure 3: Figure showing the Confusion Matrix of Question Category Classification of the Human Annotation.; Figure showing the Confusion Matrix of Question Category Classification of the Human Annotation.
Figure 4: System prompt for generating replacement columns for Ambiguous SELECT Column data generation.; System prompt for generating replacement columns for Ambiguous SELECT Column data generation.
Figure 5: System prompt for generating user clarification response for Ambiguous SELECT Column data generation.; System prompt for generating user clarification response for Ambiguous SELECT Column data generation.
Figure 6: System prompt for refining the user’s follow-up/clarification response/question.; System prompt for refining the user’s follow-up/clarification response/question.
Figure 7: System prompt for adding execution results explanation based on the SQL execution results.; System prompt for adding execution results explanation based on the SQL execution results.
Figure 8: Binary classification Prompt for data filtering. "{category_with_explanation}" will be replaced with the name and definition of the corresponding question category in Figure 10 . Few-shots examples are presented as conversation between the user and assistant in the format of message API of litellm.; Binary classification Prompt for data filtering. "{category_with_explanation}" will be replaced with the name and definition of the corresponding question category in Figure 10 . Few-shots examples are presented as conversation between the user and assistant in the format of message API of litellm.
Figure 9: Nine-way classification System Prompt. "{category_with_explanation}" will be replaced with the name and definition of four ambiguous and four unanswerable categories in Figure 10 . Few-shots examples are presented as conversation between the user and assistant in the format of message API of litellm.; Nine-way classification System Prompt. "{category_with_explanation}" will be replaced with the name and definition of four ambiguous and four unanswerable categories in Figure 10 . Few-shots examples are presented as conversation between the user and assistant in the format of message API of litellm.
Figure 10: Ambiguous and Unanswerable category and their definition. These definitions are used in the binary and 9-way classification.; Ambiguous and Unanswerable category and their definition. These definitions are used in the binary and 9-way classification.
Figure 11: Example user message and assistant message that are used as fewshots in the question category classification.; Example user message and assistant message that are used as fewshots in the question category classification.
Table showing the comparison of our work with existing datasets on ambiguity and unanswerablility in text-to-SQL task. \checkmark signifies that the category is present in the dataset. ✗ signifies that the category is not present in the dataset. * signifies that the category is defined and data is generated, but ambiguities are defined from a different perspective.
Definition and Example of Ambiguous and Unanswerable Categories. Note that the question and database schema are simplified for illustration purposes.
Dataset statistics and human annotation accuracy on 20 samples per question type. "#Ex" column shows the number of examples generated for each category. "Acc" column shows average binary classification accuracy from human expert.
Summary of Human Annotation Scores for Naturalness, Factuality, and Helpfulness.
Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Table showing examples of the ambiguous category. Here U denotes the user and A denotes the assistant.
Table showing examples of the unanswerable category. Here U denotes the user and A denotes the assistant.
Category-wise human annotation results for factuality, helpfulness, and naturalness. For every category, 1 denotes the best quality and 5 the worst.
Table showing DIN-SQL performance on ambiguous and unanswerable queries