CMPhysBench: A Benchmark for Evaluating Large Language Models in
  Condensed Matter Physics
Authors: name: Weida Wang; name: Dongchen Huang; name: Jiatong Li; name: Tengchao Yang; name: Ziyang Zheng; name: Di Zhang; name: Dong Han; name: Benteng Chen; name: Binzhao Luo; name: Zhiyu Liu; name: Kunling Liu; name: Zhiyuan Gao; name: Shiqi Geng; name: Wei Ma; name: Jiaming Su; name: Xin Li; name: Shuchen Pu; name: Yuhan Shui; name: Qianjia Cheng; name: Zhihao Dou; name: Dongfei Cui; name: Changyong He; name: Jin Zeng; name: Zeke Xie; name: Mao Su; name: Dongzhan Zhou; name: Yuqiang Li; name: Wanli Ouyang; name: Yunqi Cai; name: Xi Dai; name: Shufei Zhang; name: Lei Bai; name: Jinguang Cheng; name: Zhong Fang; name: Hongming Weng
Published: 2025-08-25 15:32:22

Abstract: We introduce CMPhysBench, designed to assess the proficiency of Large
Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark.
CMPhysBench is composed of more than 520 graduate-level meticulously curated
questions covering both representative subfields and foundational theoretical
frameworks of condensed matter physics, such as magnetism, superconductivity,
strongly correlated systems, etc. To ensure a deep understanding of the
problem-solving process,we focus exclusively on calculation problems, requiring
LLMs to independently generate comprehensive solutions. Meanwhile, leveraging
tree-based representations of expressions, we introduce the Scalable Expression
Edit Distance (SEED) score, which provides fine-grained (non-binary) partial
credit and yields a more accurate assessment of similarity between prediction
and ground-truth. Our results show that even the best models, Grok-4, reach
only 36 average SEED score and 28% accuracy on CMPhysBench, underscoring a
significant capability gap, especially for this practical and frontier domain
relative to traditional physics. The code anddataset are publicly available at
https://github.com/CMPhysBench/CMPhysBench.