Title: SPREADSHEETBENCH: Towards Challenging Real World Spreadsheet Manipulation
Benchmark: SPREADSHEETBENCH
Summary: Agentic benchmark measuring whether LLMs can do real-world spreadsheet manipulation.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Spreadsheet manipulation.
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Create a code-based solution to solve some spreadsheet instructions. Can be Python code or other code. This code is then run to execute the steps and manipulate the spreadsheet.
Task item definition: A set of instructions (taken from real-world spreadsheet questions) and initial spreadsheet.
Task source: Real task examples (e.g. GitHub issues), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 912
Dataset metadata available: Yes
Dataset metadata detail: Cell-level vs sheet-level questions.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Execute the code and evaluate exact match of table vs ground truth table.
Metric aggregation: Simple Mean
Metric statistics: Mean
Metric subscores: Yes
Metric subscores detail: Cell-level vs sheet-level manipulation
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Code Generation / 
New taxonomy: General Application / Data Analysis