Title: Benchmarking and Improving Text-to-SQL Generation under Ambiguity
Benchmark:  AmbiQT
Summary: Previous research on Text-to-SQL conversions has relied on datasets with unambiguous mappings, despite real-world queries frequently having multiple valid SQL interpretations due to schema overlaps and confusing relationships. To address this gap, the authors created AmbiQT, a benchmark featuring 3000+ examples with dual valid SQL interpretations. This  reveals that even SOTA LLMs struggle to generate all valid interpretations— because beam search algorithms produce token-level diversity rather than semantic alternatives.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Models ability to generate all valid interpretations to an ambiguous text-to-SQL query.
Phenomenon definition: It "tests performance under ambiguity in the context of current models. AmbiQT includes over 3000 examples, each associating a natural question on a database with two valid SQLs."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Ambiguity is defined as four kinds: column ambiguity, table ambiguity, join ambiguity and precomputed aggregates.
Task definition: AmbiQT tasks are natural language questions with two valid SQL solutions. The system is expexted to output all valid options in their top-k SQL outputs, for user review.
Task item definition: A natural language tasks (with two valid SQL solutions).
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 3000 tasks
Dataset metadata available: Yes
Dataset metadata detail: ambiguity type
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Response format detail: The model is prompted for its top-k answers.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean (as percentage)
Metric subscores: Yes
Metric subscores detail: by ambiguity type
Metric metascoring: EitherInTopK or BothInTopK (%)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "In this work, we curated a benchmark of ambiguous queries by perturbing SPIDER, an existing dataset. While we believe that our benchmark is a good measure of performance under ambiguity, real-life databases may exhibit more numerous as well as varied forms of ambiguity. In addition, AmbiQT only consists of examples with questions in English. Ambiguity may manifest differently based on the choice of natural language, and a corresponding study should make for interesting future work"
Taxonomy: Code Generation / Natural Language
New taxonomy: General Application / Code Generation
Other notes: The paper also includes Logical Beam which has better performance on the benchmark than the other evaluated models.