ACADREASON: Exploring the Limits of Reasoning Models with Academic
  Research Problems
Authors: name: Xin Gui; name: King Zhu; name: JinCheng Ren; name: Qianben Chen; name: Zekun Moore Wang; name: Yizhi LI; name: Xinpeng Liu; name: Xiaowan Li; name: Wenli Ren; name: Linyu Miao; name: Tianrui Qin; name: Ziqi Shu; name: He Zhu; name: Xiangru Tang; name: Dingfeng Shi; name: Jiaheng Liu; name: Yuchen Eleanor Jiang; name: Minghao Liu; name: Ge Zhang; name: Wangchunshu Zhou
Published: 2025-10-13 17:30:36

Abstract: In recent years, the research focus of large language models (LLMs) and
agents has shifted increasingly from demonstrating novel capabilities to
complex reasoning and tackling challenging tasks. However, existing evaluations
focus mainly on math/code contests or general tasks, while existing
multi-domain academic benchmarks lack sufficient reasoning depth, leaving the
field without a rigorous benchmark for high-level reasoning. To fill this gap,
we introduce the Acadreason benchmark, designed to evaluate the ability of LLMs
and agents to acquire and reason over academic knowledge. It consists of 50
expert-annotated academic problems across five high-reasoning domains,
including computer science, economics, law, mathematics, and philosophy. All
questions are sourced from top-tier publications in recent years and undergo
rigorous annotation and quality control to ensure they are both challenging and
answerable. We conduct systematic evaluations of over 10 mainstream LLMs and
agents. The results show that most LLMs scored below 20 points, with even the
cutting-edge GPT-5 achieving only 16 points. While agents achieved higher
scores, none exceeded 40 points. This demonstrates the current capability gap
between LLMs and agents in super-intelligent academic research tasks and
highlights the challenges of Acadreason.