Title: Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness
Benchmark: Dr.Spider
Summary: The paper proposes Dr Spider a text-to-SQL robustness benchmark. The authors adapt the Spider benchmark by introducing various perturbations and measuring drop in model performance. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: text-to-SQL, natural language understanding, code generation
Phenomenon defined by authors: No
Phenomenon definition: "the robustness of models with perturbations on each component of the text-to-SQL task"
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: Given a natural language query and a data base structure, the model should write a correct SQL query to obtain from the database what the NL query requests.
Task item definition: Natural language query + database structure + example of correct SQL query + the results of running the example SQL query on the content of the database
Task definition detail: The base task is described above. The "meta task" is doing this consistently among small perturbations of the problem.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 15000
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: the model generates SQL, which is then processed for grading
Metric definition: Exact Match (accuracy, F1, precision, recall), The code is executed and results are verified against ground truth results
Metric aggregation: Simple Mean
Metric statistics: No statistical methods used. just simple mean and differences in means.
Metric subscores: Yes
Metric subscores detail: Subsets are different vectors for perturbations: Perturbing the a) query semantically b) the query lexically and syntacitcally while keeping semantics invariant c) perturbing the database structure. Within each further subscores are provided.
Metric metascoring: difference between unperturbed and perturbed.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Unclear
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: No
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / Natural Language
New taxonomy: General Application / Code Generation