Title: Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Benchmark: BIRD
Summary: BIRD is a large-scale benchmark for text-to-SQL generation that focuses on realistic, noisy, and large databases. It introduces 12,751 text-to-SQL pairs over 95 databases (33.4 GB) across 37 domains, emphasizing challenges in database value comprehension, external knowledge reasoning, and SQL execution efficiency.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Database-grounded text-to-SQL generation with external knowledge reasoning and efficiency constraints
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to generate accurate and efficient SQL queries from natural language questions grounded in large, noisy, real-world relational databases, often requiring external knowledge.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a natural language question and a large relational database, generate an SQL query that retrieves the correct answer efficiently.
Task item definition: Each item includes a natural language question, an associated database, external knowledge evidence (optional), and the corresponding SQL query.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1,789
Dataset size notes: 9,428 (train) 1,534 (dev)
Dataset metadata available: Yes
Dataset metadata detail: knowledge evidence types (e.g., numeric reasoning, domain knowledge), query types (count, rank, aggregation, etc.), and database value types.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Execution Accuracy (EX) and Valid Efficiency Score (VES)
Metric definition detail: EX: Whether the predicted SQL produces the same result as the ground-truth SQL.
VES: Penalizes inefficient SQL even if correct, based on runtime efficiency relative to ground-truth.
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Metrics stratified by knowledge type (numeric, domain, value illustration) and query difficulty (simple vs complex).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Double-blind annotation procedures, SQL validity checking, external knowledge validation, and extensive error analysis performed.
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation