Title: Natural Language to Code Generation in Interactive Data Science Notebooks
Benchmark: ARCADE
Summary: A benchmark for data science tasks (natural language to code) in a Jupyter notebook. A benchmark with "realistic NL intents, rich notebook context, and a series of interrelated problems"
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Data science code generation
Phenomenon defined by authors: Yes
Phenomenon definition: Clearly defines what a computational notebook is and that the aim of the phenomenon is to generate code for the next cell that satisfies the user's intent. Scope of intended code falls within "data wrangling" or "EDA"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Generate code to fulfil the user intent for a specific cell, provided with the notebook history in-context. Code length is usually ~1-3 lines (pretty basic)
Task item definition: A natural language question specifying the user intent for the following cell.
Task definition detail: All tasks involve pandas manipulations.
Task source: Real task examples (e.g. GitHub issues), Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively), Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 1,078
Dataset metadata available: Yes
Dataset metadata detail: Data source (existing task vs newly created task)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Execution-based evaluation (unit tests)
Metric aggregation: Simple Mean
Metric statistics: Mean, error bars on figures in appendix.
Metric subscores: Yes
Metric subscores detail: Task source (existing task vs new task)
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: - All tasks use pandas, which is realistic of real DS notebooks but still not 100% coverage of the phenomenon e.g. don't consider plotting problems.
Taxonomy: Code Generation / 
New taxonomy: General Application / Data Analysis