Title: DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models
Benchmark: DA-Code
Summary: A code generation benchmark specifically for agent-based data science tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Data Science Code Generation
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Execute a data science set of instructions in an agentic setting e.g. you might need to save the results with a specific filename.
Task item definition: A set of requirements to carry out (natural language instructions) and contextual information (files in the environment and constraints on actions).
Task definition detail: The range of tasks is fairly unclear.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: Unclear how this is done.
Task ecology: Complete real task (e.g. providing medical advice to real people interactively), Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 500
Dataset metadata available: Yes
Dataset metadata detail: Difficulty level, question topic
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Unknown
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Execution-based evaluation. e.g. run the agent's code and see if it matches the ground-truth results. Plus different rubrics for each task.
Metric definition detail: The evaluation rubrics are fairly comprehensive. A strong point of the paper.
Metric aggregation: Simple Mean
Metric statistics: Mean,
Metric subscores: Yes
Metric subscores detail: Difficulty level and question topic
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Code Generation / 
Alternate taxonomy: Data Science
New taxonomy: General Application / Data Analysis