Title: FinanceMATH: Knowledge-Intensive Math Reasoning in Finance Domains
Benchmark: FinanceMATH
Summary: This paper introduces FinanceMath; a novel benchmark designed to evaluate LLMs’ capabilities in solving knowledge-intensive math reasoning problems. These problems require college-level knowledge in the finance domain for effective resolution.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Knowledge-intensive math reasoning in finance domains
Phenomenon defined by authors: Yes
Phenomenon definition: The phenomena is defned as abilities of LLMs in solving math problems requiring; 1) College-level knowledge in the finance domain 2) Interpretation of both textual and tabular data and 3) Intergration of domain-specific knowledge.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The task is defined as requiring LLMs to understand specialized financial terms, interpret tabular data to find relevant information, and then either perform step-by-step reasoning (Chain-of-Thought) or generate a structured program to solve the math question.
Task item definition: A math question containing the question text, table that the model must intepret to extract relevant numerical information, excecutable python program with solution and topic
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1000
Dataset size notes: 200
Dataset metadata available: Yes
Dataset metadata detail: Topic related to the question
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Validation
Response format: Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Simple Mean
Metric subscores: Yes
Metric subscores detail: The paper presents results across different topics and prompting strategies e.g. CoT, PoT
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: No
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Mathematics
Alternate taxonomy: Finance
New taxonomy: General Capability / Mathematical Reasoning