Title: Natural Language to Code Generation in Interactive Data Science Notebooks
Benchmark: ARCADE
Summary: A benchmark for natural language to code, specifically for data science tasks in computational notebooks, e.g. Jupyter notebooks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Data science code generation
Phenomenon defined by authors: Yes
Phenomenon definition: Given a partially completed computational notebook, the phenomenon is generating code for a specific cell that satisfies a user's intent.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a partially completed computational notebook in markdown and a user-specified intent for the following cell (in natural language), a model must generate code for the following cell. All questions require pandas manipulations.
Task item definition: A single natural language instruction requiring a 1-5 lines of code. 
Task definition detail: There is a relatively complex dependency structure between the questions because the generated code to one cell is used in-context as the notebook history when generating the code for the next cell. i.e. questions are not independent.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1,078
Dataset metadata available: Yes
Dataset metadata detail: Task origin (existing tasks vs new tasks)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Execution-based scoring.
Metric aggregation: Simple Mean
Metric statistics: Mean, error bars on some plots
Metric subscores: Yes
Metric subscores detail: Task origin (existing task vs new task)
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Only covers a subset of data science tasks e.g. not much on data visualization. 
Taxonomy: Code Generation / 
New taxonomy: General Application / Data Analysis