Title: AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries
Benchmark: AMBROSIA
Summary: Paper introduces a new benchmark dataset designed to evaluate text-to-SQL parsers' ability to handle ambiguous user requests. The dataset includes questions demonstrating scope ambiguity, attachment ambiguity, and vagueness, along with their interpretations and corresponding SQL queries. The authors highlight that existing large language models (LLMs) struggle with these ambiguities, suggesting a need for improved parser development.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: text-to-SQL parsing
Phenomenon defined by authors: Yes
Phenomenon definition: Evaluation of text-to-SQL parsers capable of recognizing and interpreting ambiguous
requests
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition:  text-to-SQL parsing, generate database, validate generated databases
Task item definition: Question, prompt, SQL query, scope/ambiguity/vagueness, generated database, score (human annotation)
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 5093
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric statistics: mean and variance
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / Natural Language
New taxonomy: General Application / Code Generation