Title: Benchmarking Data Science Agents
Benchmark: DSEval
Summary: This paper introduces DSEval, a novel bencmark for evaluating LLMs as data science agents.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Data science
Phenomenon defined by authors: Yes
Phenomenon definition: Executing a wide array of data-centric tasks, including manipulation, aggregation, visualization, and analysis, through natural language commands.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task involves responding to a natural language data science query using both the query itself and a stateful runtime session, which provides contextual information such as variables, execution history, and files. The agent must generate and return executable code that solves the query. 
Task item definition: Each problemset is represented as a Python (*.py) file with YAML syntax inside to “configure” the problem, including the query, validator configurations, execution restrictions, and external data required.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 825
Dataset metadata available: Yes
Dataset metadata detail: Difficulty
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The “Pass Rate”, which is the number of problems passed divided by all problems in the benchmark, is the default metric used to assess the quality of an agent.
Metric subscores: Yes
Metric subscores detail: By subtask dataset
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Somewhat
Author validity detail: While not explicitly addressed, the dataset construction ensures that the task items are sourced from real-world data science tasks -- which adds to its construct validity. 
Taxonomy: Code Generation / 
New taxonomy: General Application / Data Analysis