Title: InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
Benchmark: huInfiAgentDABenchEvaluatingAgents2024
Summary: This paper introduces InfiAgent-DABench, a benchmark designed to evaluate LLM-based agents on data analysis tasks. These tasks require agents to solve the tasks end-to-end by interacting with an execution environment. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Data Analysis
Phenomenon definition: Data analysis is a systematic process of examining, cleaning, transforming, and modeling data to discover useful information, inform conclusions, and support decisionmaking
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a file containing data and a question based on the data; generate executable code to answer the question and provide the final answer
Task item definition: Each item contains a question, concepts tested, constraints, answer format, file name which points to the data table and difficulty level
Task source: Real task examples (e.g. GitHub issues), 
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 257
Dataset metadata available: Yes
Dataset metadata detail: Difficulty Level, Concepts
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Response format detail: Another LLM is used in the pipeline to reformat the answer
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors define a 6-point evaluation covering suitalbleness, reasonableness, value, restrictiveness, alignment and correctness of the generated questions and conduct a human evaluation to verify each question. 85% samples are qualified and kept in final, demonstrating the effectiveness of their dataset construction method. They also compare GPT-4 generated questions to human-made ones and show that the questions written by human experts are quite similar to those generated by GPT-4.
Taxonomy: Code Generation / 
New taxonomy: General Application / Data Analysis