MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs

## Abstract

Abstract Recent large language models (LLMs) have demonstrated versatile capabilities in long-context scenarios. Although some recent benchmarks have been developed to evaluate the long-context capabilities of LLMs, there is a lack of benchmarks evaluating the mathematical reasoning abilities of LLMs over long contexts, which is crucial for LLMs’ application in real-world scenarios. In this paper, we introduce MathHay , an automated benchmark designed to assess the long-context mathematical reasoning capabilities of LLMs. Unlike previous benchmarks like Needle in a Haystack, which focus primarily on information retrieval within long texts, MathHay demands models with both information-seeking and complex mathematical reasoning abilities. We conduct extensive experiments on MathHay to assess the long-context mathematical reasoning abilities of eight top-performing LLMs.
Even the best-performing model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights the significant room for improvement on the MathHay benchmark.

## 1 Introduction

Long-context tasks arise in various applications, including summarization (Huang et al., ), multi-document question answering (Yang et al., ), prompt compression (Jiang et al., ; ), and repository-level code generation (Bogomolov et al., ).
Recent large language models (LLMs) such as GPT-4 (OpenAI, ), Claude (Claude, ), and Gemini  (Reid et al., ) have shown versatile capabilities across various long-context scenarios. They are designed to support long context modeling, being able to process up to $128$k or even $2$M tokens (Reid et al., ).

Some recent benchmarks have been developed to evaluate the long-context capabilities of LLMs.
*LongBench* (Bai et al., ) is a benchmark that covers 6 tasks, with an average length of about $7,000$ words (English version).
To evaluate the ability of LLMs to handle longer contexts, *Needle in a Haystack* (Kamradt, ) is increasingly popular.
This test requires models to locate a small, specific piece of information within varying long context windows.
However, recent advanced LLMs can easily achieve near-perfect performance on Needle in a Haystack (Dubey et al., ).
To refine the evaluation of long-context ability LLMs, several variants of the Needle in a Haystack task have been introduced.
For example, Laban et al. () presents *Summary of a Haystack*, a summarization-based test that evaluates reasoning over long contexts and the ability to grasp content importance.
*NeedleBench* (Li et al., ) positions critical data points at varying depths within texts, testing retrieval and reasoning abilities in contexts ranging from 4k to 1000k tokens.
In addition, the *BABILong benchmark* (Kuratov et al., ) is designed to test models’ reasoning across facts dispersed throughout extremely long documents, encompassing 20 tasks such as fact chaining, induction, deduction, counting, and managing lists/sets.

While these benchmarks bring complexity and diversity to evaluate the capabilities of the latest LLMs in long-context scenarios, there is still a lack of appropriate benchmarks for evaluating their long-context abilities in mathematical reasoning, which often arise in real-world situations.
For example, some example scenarios where such long-context mathematic reasoning can be helpful for users i) if there is a set of news about Nvidia’s Q2 in 2024, then the user might want to know how much revenue increased compared to the previous quarter, or the earnings per share for the quarter, and whether they exceeded analysts’ expectations ii) the user wants to compare Microsoft’s and Amazon’s cloud income and expenditure in Q2 of 2024 to help determine if they should invest in Microsoft or Amazon stocks. iii) Knowing the population growth rates for previous year and this year in a certain country can help the user decide whether to invest in real estate there in the future.
For these real-world queries, there is need for the ability to gather extensive materials from different sources, identify the precise relevant information within it and perform some mathematical reasoning in order to derive the correct answer. This inspires us to create a new mathematical reasoning benchmark to evaluate LLMs’ long-context capabilities in more real-world scenarios.

In this paper, we introduce MathHay, an automated benchmark designed to evaluate long-context mathematical reasoning in LLMs. The benchmark is built through four key stages: document collection, question generation, quality control, and haystack construction. First, we gather documents featuring real-world mathematical reasoning scenarios within a certain time period to support to form MathHay. Next, we generate four types of test tasks, varying in difficulty: (1) Single-Step, Single-Document (SSSD), (2) Multi-Step, Single-Document (MSSD), (3) Single-Step, Multi-Document (SSMD), and (4) Multi-Step, Multi-Document (MSMD). SSSD is the simplest, requiring a single relevant document and one computational step, while MSMD is the most complex, requiring multiple documents and computational steps. After question generation, we apply quality control by comparing solutions generated through different strategies to ensure high-quality data. Finally, we construct the haystack for MathHay by inserting relevant documents into noisy text using certain placement strategies. Our main contributions are summarized as follows:

- •
We introduce an automated method to create high-quality long-context mathematical reasoning benchmarks tailored for real-world scenarios within a specified time period.
- •
We present the MathHay benchmark, which includes questions of varying difficulty levels to assess LLMs’ reasoning abilities across different input lengths (32K, 64K, 128K).
- •
We conduct extensive experiments on MathHay to assess the long-context reasoning abilities of eight top-performing LLMs. Our results show that current LLMs struggle to handle mathematical reasoning tasks over long contexts, highlighting significant room for improvement on the MathHay benchmark.

## 2 Related Work

### 2.1 Long-Context Benchmarks

**Table 1: Comparative analysis of MathHay and existing long-context benchmarks.**
| Benchmark | Multi-Doc | Multi-Step | Avoidance of | Irrelevant | Realistic | Automated | Mathematical |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Tasks | Reasoning | Contamination | Documents | Documents | Construction | Reasoning |  |
| ZeroSCROLLS (Shaham et al., ) | ✓ | ✓ | × | ✓ | ✓ | × | × |
| L-Eval (Math) (An et al., ) | ✓ | × | × | × | × | × | ✓ |
| LongBench (Bai et al., ) | ✓ | × | × | ✓ | ✓ | × | × |
| BAMBOO (Dong et al., ) | × | × | ✓ | ✓ | ✓ | × | × |
| InfiniteBench (Math) (Zhang et al., ) | ✓ | ✓ | × | ✓ | × | × | ✓ |
| Loong (Wang et al., ) | ✓ | ✓ | × | ✓ | ✓ | × | × |
| NIAH (Kamradt, ) | × | × | × | ✓ | ✓ | ✓ | × |
| RULER (Hsieh et al., ) | ✓ | ✓ | × | ✓ | ✓ | ✓ | × |
| FlenQA (Levy et al., ) | ✓ | ✓ | × | ✓ | ✓ | ✓ | × |
| SummHay (Laban et al., ) | ✓ | × | × | ✓ | ✓ | × | × |
| BABILong (Kuratov et al., ) | ✓ | ✓ | × | ✓ | ✓ | ✓ | × |
| NeedleBench (Li et al., ) | ✓ | ✓ | × | ✓ | ✓ | ✓ | × |
| MathHay (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

Long-context modeling is rapidly growing, with several benchmarks developed to evaluate this capability by building on or revising existing tasks and datasets.
ZeroSCROLLS (Shaham et al., ) facilitates systematic comparisons of LLMs on tasks requiring information from long texts.
LongBench (Bai et al., ) introduces a multitask bilingual benchmark for long-context understanding, spanning 21 tasks.
Loong (Wang et al., ) highlights a key limitation of current benchmarks that artificially extend input lengths with irrelevant noise. Loong aims to reflect real-world scenarios through extended multi-document question answering. BAMBOO (Dong et al., ) addresses data contamination in long-context settings by incorporating more recent documents into the benchmark.
L-Eval (An et al., ) offers a comprehensive suite of tasks for long-context models.
InfiniteBench (Zhang et al., ) is the first benchmark featuring data lengths exceeding 100K tokens.
L-Eval and InfiniteBench include mathematical reasoning tasks, but MathHay stands out by introducing irrelevant documents, making reasoning more challenging. Needle-in-a-Haystack (NIAH) (Kamradt, ) evaluates LLM recall by embedding a fact within long contexts but focuses on shallow understanding. RULER (Hsieh et al., ) builds on NIAH with more complex tasks involving multi-hop reasoning. SummHay (Laban et al., ) focuses on summarizing large document sets, while BABILong (Kuratov et al., ) tests reasoning across dispersed facts in long documents. NeedleBench (Li et al., ) provides a customizable framework for bilingual long-context evaluations.
In addtion, DocFinQA (Reddy et al., ) is developed to assess financial reasoning in LLMs and DOCMATH-EVAL (Zhao et al., ) is manually annotated by experts to evaluate the mathematical reasoning abilities of LLMs within a context length of 35K.
Table 1 shows a comparison with existing benchmarks, MathHay is designed to automatically evaluate LLMs’ mathematical reasoning in longer, more diverse, and real-world contexts.

### 2.2 Mathematical Reasoning Benchmarks

Assessing mathematical reasoning abilities is crucial for advancing large language models. Early work in this area includes MathQA (Amini et al., ), which introduces a “large-scale” dataset of math word problems densely annotated with operation programs, curated from the AQuA (Ling et al., ) dataset.
Later, GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ) provide high-quality datasets of linguistically diverse grade school problems and challenging competition-level problems, respectively.
These datasets, known for their difficulty, are widely used to evaluate mathematical reasoning capabilities of large language models.
More recent efforts, such as LILA (Mishra et al., ), introduce a unified benchmark of $23$ mathematical reasoning tasks across multiple dimensions, further expanding the evaluation of AI systems in mathematics.
GHOSTS (Frieder et al., ) shifts the focus towards graduate-level math, addressing professional use cases for models like GPT-4 in assisting mathematicians. Our benchmark, MathHay, extends the exploration to long-context scenarios, focusing on multi-step mathematical reasoning, making it a unique contribution to benchmarking the mathematical reasoning abilities of large language models over long contexts.

## 3 Benchmark Construction

In this section, we go through the steps taken to automatically construct the MathHay benchmark and ensure the quality of the constructed benchmark.
Figure 1 illustrates the automated process, which consists of four main stages: document collection, question generation, quality control, and haystack construction.
We provide a detailed explanation of each step in this section.

Figure: Figure 1: Overview of the framework for the automatic construction of the MathHay Benchmark. The upper section illustrates the document collection process, while the lower section outlines the stages of question generation, quality control, and haystack construction.
Refer to caption: extracted/5905936/figs/framework.png

### 3.1 Document Collection

The document collection stage involves gathering texts from sources that potentially include mathematical reasoning in real-world scenarios. These documents should contain sufficient numerical values to construct data examples for the MathHay benchmark.

##### Topic Generation.

We aim for MathHay to cover diverse topics, including Financial Market Analysis, Sports Performance Metrics, and Climate Change Impact Assessment, where queries frequently require mathematical reasoning. To facilitate this, we designed a prompt to guide the LLM in generating responses on these topics. Refer to the corresponding prompt in Appendix A.1.1.

##### Relevant Document Collection.

After obtaining key topics related to mathematical reasoning, we prompt the LLM to generate subtopics along with corresponding queries. Each subtopic is paired with several specific queries. For example, under the “Nvidia’s stock price” subtopic, a potential query could be, “Compare Nvidia’s end-of-month stock prices for April 2024 and May 2024”. To ensure the queries are time-sensitive, we incorporate a time period constraint in the prompt, guiding the LLM to generate queries within a specific time range. This keeps the MathHay benchmark up-to-date and may help mitigate data leakage (test data from a benchmark might be included in the training set of newer models (White et al., )), enabling a fairer evaluation of different LLMs’ abilities. For this benchmark, we set the time period from January to August 2024.
Refer to the corresponding prompt in Appendix A.1.2.

The generated queries are used to retrieve relevant documents from online sources. For each query, we employ Tavily Search(^1^11Tavily Search is a search engine optimized for LLMs and RAG, designed for efficient, fast, and persistent results. More information is available at [https://tavily.com/](https://tavily.com/)) to gather up-to-date and relevant information. From the search results, we select the top-ranked document as the most relevant for each query.

##### Document Filtering.

After gathering the initial set of documents from search engine, we implement a filtering process to retain sufficient numerical values and informative texts for constructing high-quality mathematical reasoning problems.
First, each document has to contain more than a specific number of distinct numerical values (excluding dates) to ensure sufficient complexity for generating diverse, multi-step reasoning problems. Documents with fewer numbers might be inadequate for testing LLMs’ numerical reasoning abilities.
Second, we prioritized documents with rich context, including ample sentences, sufficient words, and diverse named entities such as people, places, and organizations. This ensured that the later generated questions could be grounded in real-world scenarios.
Through this process, we narrowed the collected documents to a refined set of high-quality documents rich in numerical values and contextual depth, enabling the generation of more realistic and challenging reasoning problems.

### 3.2 Question Generation

To construct a comprehensive benchmark for evaluating models’ capabilities in long-context mathematical reasoning, we designed a series of test tasks that vary in difficulty. The tasks can be divided into four distinct categories: (1) Single-Step, Single-Document Mathematical Reasoning Task, (2) Multi-Step, Single-Document Mathematical Reasoning Task, (3) Single-Step, Multi-Document Mathematical Reasoning Task, and (4) Multi-Step, Multi-Document Mathematical Reasoning Task.

##### Single-Step, Single-Document Mathematical Reasoning Task (SSSD).

Questions in this task require a single computational step ($+$, $-$, $\times$, $\div$) to reach the solution, based on information contained within a single document. This task assesses the model’s ability to extract relevant numerical information from a single document within a document haystack
and perform mathematical reasoning to arrive at a correct answer. The LLM is prompted to generate the question and Python solution (*i.e.*, the solution process represented as a Python program).
Refer to the corresponding prompt in Appendix A.1.3.

##### Multi-Step, Single-Document Mathematical Reasoning Task (MSSD).

This task involves questions requiring multiple computational steps to reach a solution, based on information within a single document. Unlike the SSSD, the MSSD challenges the model to identify multiple snippets containing numerical data and then correctly sequence them into intermediate reasoning steps. An LLM is used to generate the question and a one-step solution process as a Python program.

##### Single-Step, Multi-Document Mathematical Reasoning Task (SSMD).

In this category, the task requires the model to solve a problem that involves information spread across multiple documents. Although the solution involves only a single computational step, the complexity lies in the need to correctly identify and extract relevant numerical values from different documents in the haystack. The LLM is prompted to generate the question and the one-step Python solution.

##### Multi-Step, Multi-Document Mathematical Reasoning Task (MSMD).

This category represents the most complex task, challenging models to perform multi-step reasoning and extract information from multiple documents. It requires the model to sequentially process and combine numerical values from several sources, while maintaining clear mathematical reasoning and accuracy throughout the calculations. The LLM is prompted to generate the question and Python solution.

### 3.3 Quality Control

Given the range of tasks in the MathHay benchmark, which span from single-step to multi-step reasoning across one or more documents, it’s crucial that the solution process produces the correct final answer. To ensure the quality of the generated data examples, we implement a quality control process that focuses on consistency across different solutions for each question.

The quality control process begins by executing the Python solution generated by the LLM from the previous question-generation stage using a Python interpreter to get the first answer. Next, we re-feed the question and relevant documents into the LLM, prompting it to generate another Python solution. This second solution is also executed to produce a new answer. We then compare the two answers: if they match, the example is considered as high quality and suitable to be included in the benchmark. If the answers differ, the example is filtered out for being inconsistent. Refer to corresponding prompts in Appendix A.1.4.

### 3.4 Haystack Construction

Figure: Figure 2: Accuracy of GPT-4o-mini on (a) single-document; (b) two-document; (c) three-document mathematical reasoning tasks from a subset of the MathHay Benchmark, with varying relevant document placements and input lengths.
Refer to caption: extracted/5905936/figs/ana1.png

To accurately assess models’ ability to handle long-context mathematical reasoning, we construct document “haystacks” of varying sizes, simulating real-world scenarios where relevant information is buried within large volumes of irrelevant data. This setup challenges the models to filter out noise and identify the necessary details needed to solve the problem. We vary the sizes of the haystacks, with token lengths ranging from 32K to 128K tokens. Each haystack contains a mixture of documents: a small number of question-relevant documents (one relevant document for one-document reasoning tasks) and a larger pool of irrelevant ones, which are actually relevant to other unrelated queries.
These unrelated queries mainly come from different topics.
This design ensures that only a few documents in each haystack are helpful for answering the target question, making the task progressively more difficult as the haystack size increases.

We implement different placement strategies when inserting relevant documents into irrelevant documents. For single-document reasoning tasks, we experiment with three strategies: (1) First: The relevant document is placed at the beginning of the irrelevant documents, which are furthest from the target question; (2) Middle: The relevant document is inserted in the middle of the irrelevant documents; (3) Last: The relevant document is appended to the end of the irrelevant documents.

For two-document reasoning tasks, where two relevant documents are needed to solve the problem, we expand the placement strategies to combinations of positions: (1) First-First: Both relevant documents are placed at the beginning; (2) Middle-Middle: Both relevant documents are placed in the middle; (3) Last-Last: Both relevant documents are placed at the end; (4) First-Middle: One relevant document is placed at the beginning, and the second in the middle; (5) Middle-Last: One relevant document is placed in the middle, and the second at the end; (6) First-Last: One relevant document is placed at the beginning, and the other at the end.

For three-document reasoning tasks, the complexity of document placement further increases. We introduce the following four combinations: (1) First-First-First: All three relevant documents are placed at the beginning; (2) Middle-Middle-Middle: All three relevant documents are placed in the middle; (3) Last-Last-Last: All three relevant documents are placed at the end; (4) First-Middle-Last: The three relevant documents are distributed evenly, one at the beginning, one in the middle, and one at the end.

Figure 2 shows GPT-4o-mini’s accuracy on single, two, and three-document mathematical reasoning tasks, varying by document placement and input length. We can observe that the middle placement is most challenging for single-document tasks, first-middle for two-document tasks, and first-first-first for three-document tasks. Based on these results, we select these placements for each task type in constructing the MathHay benchmark.

### 3.5 Statistics of MathHay Benchmark

Figure: Table 2: Key statistics of MathHay.
Refer to caption: extracted/5905936/figs/nested_pie_chart.png

Table 2 presents the main statistics of MathHay, and Figure 3 shows the topic and task distribution of MathHay. The dataset includes $673$ questions across $10$ topics and $40$ subtopics, with $233$ single-step, $168$ two-step, and $198$ three-step reasoning tasks. On average, each question contains $33.31$ words and is linked to $1.53$ relevant documents, with the average document length being $4190.53$ tokens. The average number of reasoning steps per question is $2.00$.

The dataset is divided into verified and unverified questions. Verified data refers to questions that have been reviewed by authors to ensure the correctness of the reasoning steps. Incorrect data examples are removed, while correct ones are retained. Of the $126$ verified questions, $52$ are single-step, and their average length is $35.25$ words. These questions are linked to $1.58$ relevant documents, averaging $4139.37$ tokens in length, with $1.85$ reasoning steps per question.
The unverified portion, with $547$ questions, has an average length of $32.87$ words. These questions are linked to $1.52$ relevant documents, averaging $4202.31$ tokens, and require an average of $2.03$ reasoning steps.

## 4 Experiment

### 4.1 Experimental Setup

Models. In this work, we evaluate several cutting-edge long-context LLMs using the proposed MathHay Benchmark. Our evaluation includes both closed-source and open-source models, tested across varying token lengths: $32$K, $64$K, and $128$K.
For the closed-source models, we assess the performance of several models from the GPT series(^2^22[https://openai.com/api/](https://openai.com/api/)) (OpenAI, ; ; ), including GPT-4o (128K), GPT-4o-Mini (128K), o1-preview, and o1-mini, Claude-3.5-Sonnet(^3^33[https://claude3.pro/claude-3-5-sonnet-api/](https://claude3.pro/claude-3-5-sonnet-api/)) (Anthropic, ), and Gemini-1.5-Pro-002(^4^44[https://aistudio.google.com/app/apikey](https://aistudio.google.com/app/apikey)) (Reid et al., ).
On the open-source side, we evaluate Qwen-2.5-7B-Instruct (128K) (Team, ) and LLaMA-3.1-8B-Instruct (128K) (Dubey et al., ), two recent advanced models in the open research community.

Evaluation.
In mathematical reasoning tasks, LLMs often generate long explanations instead of directly providing numerical values as final answers. This poses challenges for traditional evaluation methods, such as rule-based or template-based exact match, which struggle to accurately assess the output. To address this, some benchmarks have adopted the practice of using LLMs as judges (Lu et al., ). Building on this, we combine rule-based exact matches with LLM judgment to assess the correctness of generated answers.
We chose GPT-4o as our evaluation judge due to its advanced reasoning and assessment capabilities (Dubois et al., ). If an exact match is achieved, the predicted answer is considered correct, and a score of 1 is assigned. In cases where the exact match fails, we rely on the LLM judge. If the LLM deems the answer correct, we also consider the predicted answer correct. Conversely, if the LLM judges the answer to be incorrect, it is marked as wrong.
A preliminary study of $100$ examples demonstrates that GPT-4o, when used as a judge, correlates almost perfectly with human evaluations in our benchmark. Detailed instructions for this evaluation process are provided in Appendix A.2.

Implement Details. We set the temperature to zero for all models to ensure deterministic predictions. For closed-source models, we use the provided API for testing. For open-source models, we use vLLM(^5^55[https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)) to build service to provide API for testing using NVIDIA A100 (40GB) GPUs.

### 4.2 Results

**Table 3: Performance of Selected Models on MathHay (32K to 128K tokens). The model with the best performance is highlighted in bold.**
| Model | Claimed | SSSD | MSSD | SSMD | MSMD | Overall |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Length | Verified | Unverified | Verified | Unverified | Verified | Unverified | Verified | Unverified | Verified | Unverified | Full |
| 32K |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaMA-3.1-8B-Instruct | 128K | 40.62 | 44.00 | 28.57 | 27.5 | 35.00 | 20.99 | 15.22 | 17.47 | 27.78 | 26.51 | 26.75 |
| Qwen-2.5-7B-Instruct | 128K | 46.88 | 52.00 | 32.14 | 27.00 | 50.00 | 34.57 | 6.52 | 21.08 | 29.37 | 30.89 | 30.61 |
| GPT-4o-mini | 128K | 71.88 | 68.00 | 42.86 | 42.50 | 50.00 | 50.62 | 26.09 | 35.54 | 45.24 | 46.25 | 46.06 |
| GPT-4o | 128K | 71.88 | 73.00 | 53.57 | 53.50 | 60.00 | 55.56 | 34.78 | 45.18 | 52.38 | 54.85 | 54.38 |
| o1-mini | 128K | 56.25 | 68.00 | 50.00 | 50.50 | 60.00 | 48.15 | 34.78 | 35.54 | 47.62 | 48.81 | 48.59 |
| o1-preview | 128K | 62.50 | 69.00 | 50.00 | 51.00 | 65.00 | 46.91 | 30.43 | 34.34 | 48.41 | 48.63 | 48.59 |
| Claude-3.5-Sonnet | 200K | 68.75 | 77.00 | 46.43 | 53.00 | 65.00 | 51.85 | 32.61 | 39.16 | 50.00 | 53.02 | 52.45 |
| Gemini-1.5-Pro-002 | 2M | 68.75 | 75.00 | 57.14 | 52.00 | 70.00 | 44.44 | 32.61 | 37.95 | 53.17 | 50.82 | 51.26 |
| 64K |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaMA-3.1-8B-Instruct | 128K | 53.12 | 58.00 | 39.29 | 30.00 | 35.00 | 24.69 | 10.87 | 19.28 | 31.75 | 31.08 | 31.20 |
| Qwen-2.5-7B-Instruct | 128K | 28.12 | 45.00 | 21.43 | 24.50 | 30.00 | 28.40 | 6.52 | 21.69 | 19.05 | 27.97 | 26.30 |
| GPT-4o-mini | 128K | 59.38 | 63.00 | 39.29 | 38.00 | 60.00 | 45.68 | 21.74 | 31.33 | 41.27 | 41.68 | 41.61 |
| GPT-4o | 128K | 65.62 | 69.00 | 53.57 | 48.50 | 65.00 | 48.15 | 32.61 | 40.96 | 50.79 | 49.91 | 50.07 |
| o1-mini | 128K | 56.25 | 60.00 | 60.71 | 47.00 | 65.00 | 50.62 | 26.09 | 33.13 | 47.62 | 45.70 | 46.06 |
| o1-preview | 128K | 59.38 | 71.00 | 42.86 | 52.00 | 65.00 | 46.91 | 28.26 | 36.75 | 45.24 | 50.09 | 49.18 |
| Claude-3.5-Sonnet | 200K | 53.12 | 67.00 | 53.57 | 50.50 | 60.00 | 48.15 | 34.78 | 36.75 | 47.62 | 49.00 | 48.74 |
| Gemini-1.5-Pro-002 | 2M | 68.75 | 73.00 | 57.14 | 53.50 | 70.00 | 50.62 | 32.61 | 38.55 | 53.17 | 52.10 | 52.30 |
| 128K |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaMA-3.1-8B-Instruct | 128K | 37.50 | 43.00 | 35.71 | 29.50 | 10.00 | 9.88 | 2.17 | 10.24 | 19.84 | 23.22 | 22.59 |
| Qwen-2.5-7B-Instruct | 128K | 15.62 | 26.00 | 14.29 | 16.50 | 20.00 | 14.81 | 10.87 | 7.23 | 14.29 | 15.17 | 15.01 |
| GPT-4o-mini | 128K | 56.25 | 65.00 | 32.14 | 39.50 | 35.00 | 39.51 | 21.74 | 30.12 | 34.92 | 41.32 | 40.12 |
| GPT-4o | 128K | 68.75 | 69.00 | 45.00 | 48.00 | 55.00 | 56.79 | 28.26 | 42.17 | 46.38 | 51.37 | 50.37 |
| o1-mini | 128K | 43.75 | 47.00 | 35.71 | 37.00 | 45.00 | 34.57 | 21.74 | 28.92 | 34.13 | 36.02 | 35.66 |
| o1-preview | 128K | 62.50 | 70.00 | 57.14 | 53.50 | 60.00 | 46.91 | 21.74 | 34.34 | 46.03 | 49.73 | 49.03 |
| Claude-3.5-Sonnet | 200K | 59.38 | 59.00 | 42.86 | 47.00 | 55.00 | 35.80 | 23.91 | 29.52 | 42.06 | 42.23 | 42.20 |
| Gemini-1.5-Pro-002 | 2M | 62.50 | 74.00 | 57.14 | 52.50 | 60.00 | 53.09 | 32.61 | 36.14 | 50.00 | 51.55 | 51.26 |

We assess eight advanced LLMs on the MathHay benchmark, with the key results presented in Table 3. GPT-4o demonstrates the highest overall performance, achieving $54.38$% at an input length of $32$K. Gemini-1.5-pro-002 achieves the highest overall performance, reaching $52.30$% at $64$K and $51.26$% at $128$K. We can see that even one of the best-performing models, Gemini-1.5-Pro-002, still struggles with long contexts, achieving only $51.26$% on the $128$K input length, which is $48.74$% below perfect accuracy. This performance gap highlights the significant room for improvement on the MathHay benchmark. In addition, to assess the quality of the automated MathHay benchmark (unverified), we computed the Spearman rank correlation between the human-verified and unverified MathHay. The resulting correlation coefficient of $0.9183$ indicates a strong alignment in model rankings across unverified and verified test data, suggesting that the automated benchmark can reliably approximate the human-verified benchmark and be useful for evaluating models.

Model Analysis: We can observe that closed-source models perform relatively well compared to open-source models across all length settings.
For instance, the best-performing open-source model, LLaMA-3.1-8B, achieves $22.59$% accuracy at $128$K. However, it still lags behind the worst-performing closed-source model, o1-mini, by $13.07$%. These findings suggest that closed-source models excel in long-context mathematical reasoning compared to open-source counterparts.

Task Analysis: From the task perspective, models consistently perform better on simpler tasks. performance of models on single-step single-document tasks (SSSD) are much better than that of models on multi-step single-document tasks (MSSD), and models on single-step multi-document tasks (SSMD) show better performance than models multi-step multi-document tasks (MSMD). For example, GPT-4o reaches $71.88$% accuracy on verified SSSD at $32$K but drops to $53.57$% on verified MSSD. Similarly, QWen-2.5-7B achieves 20.00% on verified SSMD at 128K but only $10.87$% on MSMD in the same setting. These results suggest that tasks with multiple reasoning and computational steps are significantly more challenging, especially when large amounts of noisy text are involved. Furthermore, multi-step tasks across multiple documents (MSMD) are more difficult than those within a single document (MSSD), as evidenced by consistently lower performance on MSMD across all input lengths.
This suggests that gathering information and reasoning across multiple documents is more challenging than doing so from a single document.

Length Analysis: While a few models demonstrate improved performance with longer input lengths (e.g., LLaMA-3.1-8B increases from $26.75$% at $32$K to $31.20$% at 64K), most models show a decline as input length increases. This trend suggests that longer inputs introduce more noise, limiting the ability of even advanced LLMs to accurately extract relevant information and reason effectively.

### 4.3 Analysis

##### Impact of Placement Depths and Input Lengths.

Figure: Figure 4: Performance of GPT-4o and GPT-4o-mini on single-document tasks (SSSD, MSSD) with varying placement depths and input lengths. The $y$-axis represents the depth of the relevant document. For example, $10$% depth indicates that the document is placed at the first $10$% of the input noisy text.
Refer to caption: extracted/5905936/figs/ana4.png

Figure 4 illustrates the performance of GPT-4o and GPT-4o-mini on single-document tasks with varying document placement depths and input lengths. The results show that smaller placement depths and longer input lengths lead to reduced performance, highlighting the challenge of processing relevant information that is farther from the target question among more noisy context. Notably, GPT-4o-mini demonstrates greater instability with longer input lengths, suggesting that even advanced models may struggle with extreme long inputs. These findings indicate that both insufficient context and excessive noisy text can significantly affect model robustness when handling varying input lengths and document positions.

##### Impact of the Number of Reasoning Steps.

Figure: Figure 5: Performance of models at the input length of 32K across varying reasoning steps.
Refer to caption: extracted/5905936/figs/ana2_steps.png

Figure 6 illustrates the accuracy of models with an input length of 32k across tasks requiring $1$, $2$, and $3$ reasoning steps. A common trend observed among all models is a decrease in accuracy as the number of reasoning steps increases. GPT-4o demonstrates the highest performance in handling complex multi-step tasks, followed closely by Claude-3.5-Sonnet and Gemini-1.5-Pro-002. In contrast, the other models, particularly LLaMA-3.1-Instruct and Qwen-2.5-Instruct, demonstrate steeper declines in accuracy, suggesting they are less adept at handling tasks that require multiple reasoning steps.

##### Impact of Time Period

We aim to assess whether documents collected from queries over different years may impact performance, particularly to explore if more recent documents pose a greater challenge due to potential contamination avoidance. Figure 6 shows the model performance across single-document tasks (SSSD, 2SSD, 3SSD) from 2021 to 2024. While 2SSD and 3SSD display a gradual performance decline over time, SSSD remains relatively stable. This indicates that the time period could influence model accuracy, but the effect is uncertain and not clearly confirmed by the current analysis. Further experiments are needed to investigate this hypothesis more thoroughly.

##### Analysis of Models across Topics and Tasks.

Figure: Figure 7: Model performance on the MathHay benchmark at $32$K across different topics and tasks.
Refer to caption: extracted/5905936/figs/ana3.png

Figure 7 compares the performance of eight models on the MathHay benchmark across (a) different topics and (b) different tasks. In Figure 7(a), GPT-4o generally outperforms the other models across most topics, such as SCM and HDA, with the largest coverage. Claude-3.5-Sonnet and Gemini-1.5-Pro-002 perform similarly but fall behind GPT-4o. LLaMA3.1 and Qwen2.5 perform noticeably lower across all topics. In Figure 7(b), GPT-4o also excels across various tasks, particularly in the SSSD and 3S2D (Three-Step, Two-Document) tasks, maintaining strong accuracy. The smaller models—GPT-4o-mini and o1-mini show similar trends but generally underperform relative to GPT-4o. Again, LLaMA3.1 and Qwen2.5 struggle, especially in the multi-step tasks (e.g., 3SSD, 3S2D, and 2D2D), further indicating their difficulty in handling complex reasoning. Overall, GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro-002 demonstrate superior robustness across both different topics and tasks, while open-source models show much weaker performance, particularly in complex, multi-step tasks.

## 5 Conclusion

In this work, we introduced MathHay, a benchmark specifically designed to evaluate the long-context mathematical reasoning abilities of LLMs. MathHay is built to challenge LLMs with real-world scenarios requiring both complex reasoning and numerical computation across varying input lengths and document depths. The experimental results show that while Gemini-1.5-Pro-002 performs the best, achieving $51.26$% accuracy on tasks with input lengths up to 128K tokens, there remains a substantial performance gap, indicating significant room for improvement. Our findings further reveal that open-source models struggle considerably compared to closed-source counterparts, particularly in tasks that require multi-step reasoning over multiple documents. This underscores the challenges that LLMs face when dealing with noisy and irrelevant information in long contexts, making MathHay a crucial benchmark for driving future advances in long-context mathematical reasoning. MathHay also offers a novel and automated framework for constructing benchmark datasets, with strong correlations between human-verified and unverified data. This automation enables scalable and efficient testing for future LLMs. MathHay aims to drive the development of models with enhanced reasoning capabilities for complex, real-world mathematical tasks.

## Appendix A Prompt Examples

All prompts used in MathHay construction consist of two key components: a prompt template and an output parser. The output parser enables users to define any Pydantic model and query LLMs for outputs that adhere to the specified schema.

### A.1 Prompts Used in MathHay Construction

#### A.1.1 Prompt for Topic Generation

Figure: Figure 8: Example prompt for asking the LLM to generate 10 topics.

#### A.1.2 Prompt for Subtopic and Query Generation

Figure: Figure 9: Example prompt for asking the LLM to generate subtopics and corresponding queries.

#### A.1.3 Prompt for SSSD Question Generation

Figure: Figure 10: Example prompt for generating Single-Step Single-Document (SSSD) questions using an LLM. Similar prompts are used for tasks like Multi-Step Single-Document (MSSD), Single-Step Multi-Document (SSMD), and Multi-Step Multi-Document (MSMD).

#### A.1.4 Prompt for Quality Control

Figure: Figure 11: Example prompt for asking the LLM to Python Solution When Given question, relevant quantities, and relevant documents.

### A.2 Prompt for Evaluation

Figure: Figure 12: Example prompt for asking the LLM to judge if two solutions are the same.

### A.3 Prompt for Solving Problems in MathHay

Figure: Figure 13: Example prompt for asking the LLM to solve the problems in MathHay.

## Appendix B Test Data Examples from MathHay

### B.1 Example of Data for SSSD

Figure: Figure 14: Illustrative Examples of data for the Single-Step Single-Document (SSSD) task.

### B.2 Example of Data for MSSD

Figure: Figure 15: Illustrative Examples of data for the Single-Step Single-Document (MSSD) task.

### B.3 Example of Data for SSMD

Figure: Figure 16: Illustrative Examples of data for the Single-Step Single-Document (SSMD) task.

### B.4 Example of Data for MSMD

Figure: Figure 17: Illustrative Examples of data for the Single-Step Single-Document (MSMD) task.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - I’m curious if the topics and tasks affect the mathematical reasoning performance of LLMs?
- In the classic Needle-in-a-Haystack paper (Kuratov et al., 2023), a pattern emerged where models struggled with retrieval when the ‘needle’ appeared near the beginning but not in the first sentence. However, I haven’t observed any similar pattern in mathematical reasoning—only a general decline in performance as the context window expands, regardless of the needle’s position. This is expected and not particularly insightful. Do you have any educated guesses as to why?”
soundness: 3
strengths: - relatively written paper, easy to follow
- an automated and scalable way to construct a Denmark 
- I like the way of using time-sensitive Q&As to mitigate the contamination problem
- the investigation is the impact of time period is interesting, which might motivate further research about the data contamination problem in LLMs.
summary: In this paper, the authors introduce MathHay, a benchmark and data collection pipeline designed to evaluate the mathematical reasoning capabilities of LLMs over extended contexts. Unlike existing benchmarks, MathHay incorporates varied context lengths and introduces multiple documents (ranging from one to three) to assess the models’ ability to filter relevant information from noise. Additionally, it includes multi-step tasks (up to three arithmetic steps) to measure reasoning depth. The benchmark also spans a range of topics, from financial market analysis to agricultural economics. Leading LLMs, including GPT-4, o1-preview, and Gemini-1.5-Pro-002, have been tested, with empirical results indicating that there is still room for improvement in current models.
weaknesses: - More quantitative and qualitative evaluation may be needed on the quality of the automated generated corpus.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: 1.	Conduct Experiments on Noise Impact: Add experiments that analyze how the model's reasoning ability is affected as the amount of irrelevant (noise) documents gradually increases. This can provide a more nuanced understanding of how noise impacts long-context reasoning performance.
2.	Incorporate Diverse Reasoning Scenarios: Design experiments that assess different types of long-context reasoning skills (e.g., logical deductions, multi-step calculations) across varied noise levels, ensuring that the benchmark evaluates both retrieval and reasoning capabilities in a balanced way.
3.	The paper only introduces four specific combinations for placing three relevant documents in reasoning tasks (First-First-First, Middle-Middle-Middle, Last-Last-Last, and First-Middle-Last). Why doesn’t it explore all possible combinations of placements?
soundness: 2
strengths: 1.	MATHHAY employs an automated quality control and evaluation process, combining exact match and LLM-based assessment, which achieves high correlation with human evaluations. This automation facilitates scalable and efficient testing for future LLM development.
2.	The study provides in-depth insights into how factors like document placement depth, input length, reasoning step count, time period, and topic influence model performance. This multi-angle approach highlights specific challenges for LLMs and provides a nuanced view of model capabilities and limitations.
summary: This paper presents MATHHAY, an automated benchmark developed to evaluate the long-context mathematical reasoning capabilities of large language models (LLMs). The benchmark construction involves four main stages: document collection, question generation, quality control, and haystack construction. Real-world mathematical reasoning documents are gathered, and four types of tasks with increasing complexity are generated—ranging from single-step, single-document tasks to multi-step, multi-document tasks. A quality control process ensures data reliability by verifying solution consistency, and a haystack structure is created to challenge models with relevant documents embedded in noisy text. The contributions include: (1) a method to automatically generate a high-quality benchmark for long-context mathematical reasoning, (2) the introduction of MATHHAY with tasks of varying difficulty levels to evaluate reasoning abilities across different input lengths, and (3) experimental findings showing that existing LLMs face significant challenges in handling mathematical reasoning over long contexts, indicating room for improvement in this area.
weaknesses: 1.	The quality control process relies on consistency across multiple solutions generated by the model, but this does not fully ensure mathematical correctness. Consistent solutions may still be incorrect, resulting in potential "false positives" in data quality. Without manual verification or deeper validation, the benchmark may include examples that lack rigor, reducing its reliability for assessing true mathematical reasoning abilities.
2.	This approach relies heavily on adding noise and complex placements to increase task difficulty, which may primarily test the model’s noise filtering rather than its core mathematical reasoning abilities. The setup could overly focus on retrieval challenges, potentially overlooking deeper aspects of mathematical understanding and reasoning.
3.	The benchmark lacks clarity on ensuring that multi-step tasks genuinely require multiple steps. If questions can be solved with either single-step or multi-step solutions, it may lead to inconsistent categorization.
4.	The benchmark does not address how it ensures that generated questions are genuinely related to document content. If the model generates irrelevant or “hallucinated” information, it could compromise the quality and reliability of the generated task.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.Will the benchmark be updated in the future? 
2. How did you do to filter documents? How  filtering step numerical content  is done?
3. When consistent answers were incorrect?
soundness: 2
strengths: 1. Proposed benchmark in thie paper is tailored  with varying context length, reasoning steps and spectrum noisy or irrelevant content in document to measure  comprehensive performance in LLMs
2. existing method in mathmatical reasoning are difficult to achieve good performance, which reflects the necessity of this benchmark and the high difficulty of mathematical reasoning and information extraction.
summary: The paper proposes a method to generate long-context mathematical reasoning benchmarks for evaluating LLMs' capabilities of information extraction and mathematical reasoning under certain long-context noisy and question-irrelavant circumstance. This paper also evaluates the performance of several LLMs on proposed benchmark MathHay.
weaknesses: 1. Proposed methodology could not be considered as a contribution.The Architecture shown in Figure 1 is not novel while the process is well represented.
2. No quantitative description of the quality of the answers to the questions is given in quality control pipline.
3. The article does not provide specific details of the reasoning steps.
4. The experimental part is not detailed enough, only describing different methods, without showing the connection between the benchmark and existing methods. Therefore, experiment is too simple to be a contribution in this paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * How does this method ensure the alignment between the question and the Python solution? 
* Concern about how to cover if a question has multiple correct solutions, especially based on the quality control strategy?
* Is there an evaluation method to check if the generation question meets the SSSD/MSSD/SSMD/MSMD tasks? 
* The evaluation relies on GPT-4o performance, which is not sound enough. Moreover, 100 examples is too small scale for human evaluation. 
* In Table 2, the verified two-step questions and three-step questions are 0. Could you please explain this? And what are the standards for selecting the 126 verified questions?
soundness: 2
strengths: * The paper presents a novel approach to evaluating long-context mathematical reasoning in LLMs, addressing a gap in existing benchmarks. 
* The paper is well-structured and clearly written. The methodology is explained in detail, and the experimental setup and results are presented in a way that is easy to follow.
* Experiments are conducted on 8 top-performing LLMs. Rich analyses are included.
summary: This paper introduces an automated benchmark MATHHAY for accessing the long-context mathematical reasoning capabilities of LLMs. Unlike previous benchmarks like Needle in a Haystack, which focuses primarily on information retrieval within long texts, MATHHAY demands models with both information-seeking and complex mathematical reasoning abilities. Experiments are conducted on 8 top-performing LLMs.
weaknesses: Please see the questions below.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: Will the benchmark become obsolete quickly? Will you be generating new versions frequently?

Did you use LLMs to filter documents? Some numerical content may be in words, does your filtering step account for that?

When you manually evaluate the answers, what percent of the consistent answers were incorrect? There is a significant gap in the performance on the verified and unverified sets. Also, the key statistics indicate that all the verified questions were single step, so how were the verified stats generated for MSSD and MSMD?

Some typos or unclear statements.
line 51: mathematic reasoning

line 78: within a certain time period to support to form

Page 19 - The documents have strange substrings like 0̆0bb and Tesla2̆019s. Perhaps a copy/paste or LaTeX error?

Line 1065: Figure 15: Examples of data for the Single-Step Single-Document (MSSD) task.

Line 1121: Figure 16: Examples of data for the Single-Step Single-Document (SSMD) task.

Line 1171: Figure 17: Examples of data for the Single-Step Single-Document (MSMD) task.
soundness: 3
strengths: The benchmark and the methodology to generate it are clearly presented.
The experiments show that the benchmark has sufficient value for helping evaluate future LLMs.
summary: The paper presents a new benchmark for numerical reasoning in long contexts. The benchmark consists of 673 numerical questions, and each question is associated with a set of documents. Some questions can be answered by examining only one document from the set, others need information from multiple documents. Some questions need one arithmetic operation, others need up to 3 steps.

The authors generated the benchmark data semi-automatically, and then evaluated various LLMs with it. They point out that no LLMs were able to answer more than half the questions correctly, leading them to the conclusion that this benchmark would be useful in tracking the capabilities of LLMs.
weaknesses: The benchmark is rather small (673 questions), only 20% of which were verified by humans. So in effect, LLMs are evaluating other LLMs!

While the paper repeatedly talks about mathematical reasoning, the benchmark only tests very simple numerical calculations. It does not test true mathematical ability, such as solving problems in a math textbook. Documents are selected if they contain many numbers, as the questions are based on numbers.

The methodology for generating multi-step questions was not described clearly. Since the questions are machine generated, they tend to be rather trivial, and sometimes meaningless, as shown in the few examples in the Appendix.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: - Please clarify the differences with InfiniteBench and L-Eval which are the two benchmarks with long contexts and mathematical reasoning.

- What does make your methodology a contribution and the generated benchmark a “unique contribution” to the area?

- Give more details about document filtering, in particular the parameters used (number of distinct numerical values, etc.)

- What do you precisely mean by “high-quality document”? What is the metric used to evaluate it?

- How do you evaluate the relevance of the questions?

- The benchmark is said to be reviewed to ensure the correctness of the reasoning steps, however they are not explicit in the answer. Please explain how this is done.

Other comments

Revise and update references including published versions.

The sentence in Pg. 3, L. 132 “Our benchmark … making it a unique contribution … “ does not appear to be clearly justified.
soundness: 1
strengths: In general, the paper is very well written. In particular, the introduction reads nicely and the goals are clearly and precisely stated. 

Looking for an automated methodology for generating benchmarks.

The partition of the difficulty of the tasks with varying number of operations and number of involved documents.
summary: The paper proposes an LLM-based methodology for generating mathematical benchmarks, composed of a question in some topic which involves a numeric answer which has to be calculated from a text provided as context to the question, together with the expected answer. It also generates a benchmark and evaluates the performance of several LLMs on it.
weaknesses: Related work reads like a survey but it is not clear the actual relation of the contribution of the paper with the cited work. In particular, the contribution with respect to InfiniteBenchAlmost all bibliographic references are from arXiv while there are (newer) published versions of some of the papers.

There is no clear evidence that the actual methodology could be considered to be a contribution. The Architecture shown in Figure 1 is reasonable but it does not seem to be a breakthrough. Several parts of the process are described in very vague way, in particular “Document filtering” which is somewhat critical.

The proposed methodology  does not provision any mechanism for evaluating the quality of the generated questions. The quality control phase only mentions the evaluation of the answers with respect to the questions.

The generated problems does not seem to contain reasoning steps but only the question and the answer.

Experiments focus on evaluating the performance of LLM on the constructed benchmark rather than on evaluating the relevance of the methodology and the benchmark themselves, which are claimed to be the main contribution.

Therefore, there is a lack of adequate experimental evaluation to assess the performance of the methodology to generate relevant benchmarks.