Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics
  Research Benchmark
Authors: name: Minhui Zhu, name: Minyang Tian, name: Xiaocheng Yang, name: Tianci Zhou, name: Penghao Zhu, name: Eli Chertkov, name: Shengyan Liu, name: Yufeng Du, name: Lifan Yuan, name: Ziming Ji, name: Indranil Das, name: Junyi Cao, name: Yufeng Du, name: Jinchen He, name: Yifan Su, name: Jiabin Yu, name: Yikun Jiang, name: Yujie Zhang, name: Chang Liu, name: Ze-Min Huang, name: Weizhen Jia, name: Xinan Chen, name: Peixue Wu, name: Yunkai Wang, name: Juntai Zhou, name: Yong Zhao, name: Farshid Jafarpour, name: Jessie Shelton, name: Aaron Young, name: John Bartolotta, name: Wenchao Xu, name: Yue Sun, name: Anjun Chu, name: Victor Colussi, name: Chris Akers, name: Nathan Brooks, name: Wenbo Fu, name: Christopher Wilson, name: Jinchao Zhao, name: Marvin Qi, name: Anqi Mu, name: Yubo Yang, name: Allen Zang, name: Yang Lyu, name: Peizhi Mai, name: Xuefei Guo, name: Luyu Gao, name: Ze Yang, name: Chi Xue, name: Dmytro Bandak, name: Yaïr Hein, name: Yonatan Kahn, name: Kevin Zhou, name: John Drew Wilson Jarrod T. Reilly, name: Di Luo, name: Daniel Inafuku, name: Hao Tong, name: Liang Yang, name: Ruixing Zhang, name: Xueying Wang, name: Ofir Press, name: Nicolas Chia, name: Eliu Huerta, name: Hao Peng
Published: 2025-09-30 17:34:03

While large language models (LLMs) with reasoning capabilities are
progressing rapidly on high-school math competitions and coding, can they
reason effectively through complex, open-ended challenges found in frontier
physics research? And crucially, what kinds of reasoning tasks do physicists
want LLMs to assist with? To address these questions, we present the CritPt
(Complex Research using Integrated Thinking - Physics Test, pronounced
"critical point"), the first benchmark designed to test LLMs on unpublished,
research-level reasoning tasks that broadly covers modern physics research
areas, including condensed matter, quantum physics, atomic, molecular & optical
physics, astrophysics, high energy physics, mathematical physics, statistical
physics, nuclear physics, nonlinear dynamics, fluid dynamics and biophysics.
CritPt consists of 71 composite research challenges designed to simulate
full-scale research projects at the entry level, which are also decomposed to
190 simpler checkpoint tasks for more fine-grained insights. All problems are
newly created by 50+ active physics researchers based on their own research.
Every problem is hand-curated to admit a guess-resistant and machine-verifiable
answer and is evaluated by an automated grading pipeline heavily customized for
advanced physics-specific output formats. We find that while current
state-of-the-art LLMs show early promise on isolated checkpoints, they remain
far from being able to reliably solve full research-scale challenges: the best
average accuracy among base models is only 4.0% , achieved by GPT-5 (high),
moderately rising to around 10% when equipped with coding tools. Through the
realistic yet standardized evaluation offered by CritPt, we highlight a large
disconnect between current model capabilities and realistic physics research
demands, offering a foundation to guide the development of scientifically
grounded AI tools.