Title: DOCMATH-EVAL: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents
Benchmark: DOCMATH-EVAL
Summary: Introduces DOCMATH-EVAL, a benchmark for assessing the ability of LLMs to extract information from complex financial documents, and combining it in complicated mathematical formulas. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: LLMs’ numerical reasoning in real-world scenarios, particularly in specialized fields such as finance, medicine, and science. These expert domains necessitate LLMs to interpret complex, domain-specific documents, applying numerical reasoning to complex problem-solving.
Phenomenon defined by authors: No
Phenomenon definition: LLMs’ numerical reasoning in real-world scenarios, particularly in specialized fields such as finance, medicine, and science. These expert domains necessitate LLMs to interpret complex, domain-specific documents, applying numerical reasoning to complex problem-solving.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Presented with a numerical reasoning question q and a financial document consisting of textual contents E and structured tables T , the task is to generate the numericvalue answer a:  ˆa = arg max PLM(a | q, E, T ) (1)
Task item definition: [System Input]:
You are a financial expert, you are supposed to answer the given question based on the provided financial document context. You need to first think through the problem step by step, documenting each necessary step. Then you are required to conclude your response with the final answer in your last sentence as "Therefore, the answer is {final answer)".
The final answer should be a numeric value.
[User Input]:
{Document context)
Question: (question}
Let's think step by step to answer the given question.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 3200
Dataset size notes: validation: 800
Dataset metadata available: Yes
Dataset metadata detail: Split into 4 difficulty levels
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Split by 4 difficulty levels
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning