MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
  in LLMs
Authors: name: Lei Wang; name: Shan Dong; name: Yuhui Xu; name: Hanze Dong; name: Yalu Wang; name: Amrita Saha; name: Ee-Peng Lim; name: Caiming Xiong; name: Doyen Sahoo
Published: 2024-10-07 02:30:07

Abstract: Recent large language models (LLMs) have demonstrated versatile capabilities
in long-context scenarios. Although some recent benchmarks have been developed
to evaluate the long-context capabilities of LLMs, there is a lack of
benchmarks evaluating the mathematical reasoning abilities of LLMs over long
contexts, which is crucial for LLMs' application in real-world scenarios. In
this paper, we introduce MathHay, an automated benchmark designed to assess the
long-context mathematical reasoning capabilities of LLMs. Unlike previous
benchmarks like Needle in a Haystack, which focus primarily on information
retrieval within long texts, MathHay demands models with both
information-seeking and complex mathematical reasoning abilities. We conduct
extensive experiments on MathHay to assess the long-context mathematical
reasoning abilities of eight top-performing LLMs. Even the best-performing
model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
the significant room for improvement on the MathHay benchmark.