Title: BizBench: A Quantitative Reasoning Benchmark for Business and Finance
Benchmark: BizBench
Summary: This paper introduces BizBench, a benchmark for evaluating models’ ability to reason about realistic financial problems. BizBench comprises eight quantitative reasoning tasks, focusing on question answering (QA) over financial data via program synthesis.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Financial quantitative reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: This paper proposes a benchmark for evaluating models’ ability to reason about realistic financial problems as the ability to perform question-answering over structured and unstructured financial data.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: BizBench consists of three interrelated types of tasks for assessing transparent and accurate financial reasoning: program synthesis, quantity extraction, and domain knowledge.
Task item definition: The benchmark comprises of three separate sub-tasks. The task items for each sub-task are described below;
- Program Synthesis: Each example contains a natural language question, optionally text or structured data source, and a Python program that produces a numeric answer to the question
- Quantity Extraction: A document snippet and a target label as input, the expected output is the quantity span from the snippet corresponding to the label
- Domain Knowledge: MCQA and function stub including a docstring and type hints for code completion. 
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 5,448
Dataset size notes: 14,394
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice, Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Scores are provided for each sub-task, sub-task dataset, and number of few-shot examples provided
Metric access: Outputs alone
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Somewhat
Author validity detail: The authors attempt to demonstrate construct validity by stating that the questions used in the benchmark "are written by financial professionals using real-world data and financial knowledge. As such, they are closer to the kinds of questions that business and financial professionals answer as part of their workflows." However, they do not empirically validate this with any extensive experiments. 
Taxonomy: Finance / 
New taxonomy: Specific Application / Profesisonal Domains