MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
  in LLMs
Authors: name: Lei Wang, name: Shan Dong, name: Yuhui Xu, name: Hanze Dong, name: Yalu Wang, name: Amrita Saha, name: Ee-Peng Lim, name: Caiming Xiong, name: Doyen Sahoo

Recent large language models (LLMs) have demonstrated versatile capabilities
in long-context scenarios. Although some recent benchmarks have been developed
to evaluate the long-context capabilities of LLMs, there is a lack of
benchmarks evaluating the mathematical reasoning abilities of LLMs over long
contexts, which is crucial for LLMs' application in real-world scenarios. In
this paper, we introduce MathHay, an automated benchmark designed to assess the
long-context mathematical reasoning capabilities of LLMs. Unlike previous
benchmarks like Needle in a Haystack, which focus primarily on information
retrieval within long texts, MathHay demands models with both
information-seeking and complex mathematical reasoning abilities. We conduct
extensive experiments on MathHay to assess the long-context mathematical
reasoning abilities of eight top-performing LLMs. Even the best-performing
model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
the significant room for improvement on the MathHay benchmark.